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I. INTRODUCTION 


A. THE MOTIVATION TO ACCURATELY MEASURE THE EFFECTIVENESS 

OF INFORMATION SYSTEMS 

After spending billions of dollars on automating the 
offices of white collar workers, organizations find that 
they must scrutinize the claims by computer vendors about 
increased productivity. To prudently manage a resource an 
organization must have a means to measure its benefits as 
well as its costs. 

An appropriate managerial question that arises is 
whether an organization is getting a return on its 
investment ina resource. This necessity is not diminished 
because the resource in question is an information system. 
Managers of information systems are pressed by senior line 
management to answer, again and again, the question, "What 
is the value of the company’s investment in information 
systems?" (Matlin, 1979). 

According to Strassman (1976), "Industries and 
governments all over the world are currently struggling to 
contain rising administrative and clerical overheads by 
automating information handling in the office." 

While the clerical work force has been the fastest 
growing segment of the labor force in nearly every 


industrialized country since the 1960s, the productivity of 


this sector has been flat since the early 1970s (Harris, 
1987). This has been the result despite the fact that 40% 
of capital spending (in the United States) has been directed 
at information technology. 

General managers in the United States, when surveyed by 
the University of Minnesota Management Information Systems 
Research Center, ranked the measuring of system 
effectiveness as the fourth most critical problem they face 
in managing information systems (Brancheau and Wetherbe, 
1987). Sixty percent of the management information systems 
executives polled in the same survey placed the measuring of 
effectiveness in their "top ten" list of critical problems. 
The assessment of system success has been in the top ten 
list of critical management information systems problems 
Since it was first formulated in 1980. 

Information systems practitioners are clearly expressing 
considerable concern over the effectiveness of information 
systems. Consequently, researchers are devoting 
considerable efforts to studying effectiveness of systems. 
Professional journals routinely publish articles reporting 
researchers’ findings concerning effectiveness of 
information systems; however, are these findings based upon 
measures that have demonstrated validity? 

B. ARE RESEARCHERS ADDRESSING VALIDITY ISSUES? 

Evaluations regarding the effectiveness of information 


systems are based upon measures collected and analyzed by 


information systems researchers. Such measures are gathered 
using a variety of data collection techniques. Researchers 
in sociology, such as Campbell and Stanley, Denzin, Phillips 
and Suchman, have demonstrated that data collection 
techniques significantly influence data validity. 
C. THE RESEARCH QUESTION 

This study asks if information systems researchers, who 
reported measures of information systems effectiveness, 
sufficiently addressed validity issues arising from data 
collection techniques. 

Subsidiary research questions follow: 

1. What data collection instruments do researchers use? 


2. What do researchers purport to measure with the 
instruments? 


3. How do researchers administer the instruments? 


4. How and to what extent do researchers address validity 
issues? 


Using these research questions, the study will assess the 
general awareness and sensitivity of researchers to validity 
issues. 
D. SCOPE OF THE STUDY 

This study consists of an analysis of published 
empirical methods, since 1970, which were used to evaluate 
effectiveness of information systems. The analysis 
identifies how researchers collected data when evaluating 
the effectiveness of information systems and how they 


subsequently addressed validity issues. Using literature 


from sociology, the data collection techniques used by 
information systems researchers are examined and their 
impact upon data validity is discussed. 
E. ASSUMPTIONS 

It is assumed that the 37 articles analyzed in this 
study represent the current state of information systems 
research. Findings regarding the researchers who wrote 
these articles are assumed to apply to information systems 
researchers in general. 
F. ORGANIZATION OF THE STUDY 

This study is organized into introduction, literature 
review, methodology, findings, and conclusions chapters. 
The literature review chapter explains the concept of 
validity, drawing primarily upon works of prominent social 
scientists. The methodology chapter details the selection 
of studies for review and the methods in which these studies 
were analyzed. The findings chapter presents data collected 
during the analysis of 37 articles regarding effectiveness 
of systems. The conclusions chapter discusses the 
significance of key findings and provides recommendations 
for enhancing the credibility of information systems 


research. 


II. LITERATURE REVIEW 


A. CLASSICAL DEFINITION OF MEASUREMENT 
Stevens (1951) provides a popular definition of 
measurement, "Measurement is the assignment of numbers to 
objects or events according to rules." This definition is 
appropriate for the physical sciences, where the objects and 
events are the focus of research. Information systems 
researchers often measure phenomena which are typically too 
abstract for characterization as either objects or events. 
Information systems researchers are currently studying 
issues of effectiveness, benefit, and success of information 
systems. All of these issues are too abstract for direct 
observation and quantification. 
B. MEASUREMENT AND THE SOCIAL SCIENCES 
Social science researchers study phenomena which are 
equally abstract. They have adopted an alternative 
definition of measurement, as suggested by Blalock (1968), 
"Sociological theorists often use concepts that are 
formulated at rather high levels of abstraction. These 
are quite different from the variables that are the 
stock-in-trade of empirical sociologists. . . . The 
problem of bridging the gap between theory and research 
is then seen as one of measurement error." 
Carmines and Zeller (1979) view measurement most usefully as 


the process of linking abstract concepts to empirical 


indicators. 


Carmines and Zeller’s definition of measurement in 
social science research identifies a process that involves 
both theoretical as well as empirical considerations. They 
further elaborate, 
"From an empirical standpoint, the focus is on the 
observable response--whether it takes the form of a mark 
on a self-administered questionnaire, the behavior 
recorded in an observational study, or the answer given 
to an interviewer. Theoretically, interest lies in the 
underlying unobservable (and directly unmeasurable) 
concept that is represented by the response. : 
Measurement focuses on the crucial relationship between 
the empirically grounded indicator(s)--that is, the 
observable response--and the underlying unobservable 
concept (s)." 

C. MEASUREMENT AND INFORMATION SYSTEMS 

To deal with the abstract nature of information systems 
issues, many information systems researchers are using the 
measurement process and data collection techniques commonly 
found in social science. Information systems researchers 
often use observable response, unobservable concepts, and 
linking relationships to measure information systems 
phenomena. Figure 1 illustrates how the measurement process 
used by many information systems researchers mirrors the 
measurement process used in social science. Consequently, 
these information systems researchers have, consciously or 


unconsciously, adopted the measurement process of social 


science. 


EMPIRICALLY GROUNDED 
INDICATORS and OBSERVABLE RESPONSE 


| User UNOBSERVABLE 
Satisfaction | CONCEPT 


Questionnaires 


oa eae System 

P Effectiveness 
a | de 
Usage 


Observations 
Records and documents 





FIGURE 1. Information Systems’ Measurement Process 

Social science researchers have long recognized that the 
validity of the measurement process is critical to deriving 
useful information from any collected data. Since the same 
measurement process is now used by information systems 
researchers, they are now logically subject to the same 
validity considerations. Information systems researchers 
who use this form of measurement must understand the concept 
of validity. 
D. CONCEPT OF VALIDITY 

Suchman (1967) discusses validity, reliability, their 
relationship, and its importance. Validity refers to the 
degree to which any measure or procedure succeeds in doing 
what it is purported to do. Closely related to validity is 
the concept of reliability. Reliability refers to the 
degree to which any measure or procedure can be depended on 
to secure consistent results upon repeated application. 
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Validity and reliability are interdependent. There can 
be no validity without reliability. When a measure is shown 
to be valid, its reliability is taken for granted. However, 
one can have high reliability without validity. The 
demonstration of a measure’s reliability does not establish 
its validity. 

Validity is the most important criterion for evaluating 
a measuring instrument. Every phase of the evaluation 
process is influenced by its validity. Validity concerns 
the formulation of objectives, the collection of data, and 
the interpretation of findings. The validity of an 
evaluation study refers not only to the validity of its 
specific criteria or measures, but also to the theory 
underlying the formulation of the hypotheses concerning the 
relationship of the activities to the objective. 

E. TYPES OF VALIDITY 

Phillips (1971) states that to determine the validity of 
a measuring instrument, one must evaluate it against 
relevant or appropriate criteria. Social science 
researchers use numerous criteria to determine the validity 
of measuring instruments. 

The following are types of validity and thus used as 
criteria to verify the validity of measuring instruments: 

1. Content validity is the making of a judgement as to 
the representativeness and comprehensiveness of the 
measuring instrument’s indicators constituting a 
measurement (Phillips, 1971). Content validity refers 


to the ability of the instruments to measure what it 
purports to measure. 


Face validity is an aspect of content validity. It is 
the extent to which an instrument seems to measure 
what it is supposed to measure. Face validity is the 
obvious significance of the measure. It is usually a 
judgment made by the designer of an instrument, after 
its completion (Phillips, 1971). 


Consensual validity is a face validity which uses a 
panel of experts to develop a consensus concerning the 
validity of a measure (Suchman, 1967). 


Criterion or correlational validity correlates a 
measure with another indicator that one "knows" 
measures what is being sought. The new measure is 
correlated with an existing, accepted measure, thus 
establishing the validity of the new measure. High 
correlation of two measures demonstrates the validity 
of the new measures (Suchman, 1967). 


Concurrent validity is a type of criterion validity 
that refers to the correlation of a measure and a 
criterion at the same point in time. For example, 
correlating data gathered through exit polling 
interviews to official voting records (Carmines and 
Zeller, 1979). 


Predictive validity is another type of criterion 
validity that refers to the correlation of present 
measures with behavior that is predicted to take place 
in the future (Suchman, 1967). For example, 
correlating the results of an undergraduate’s SAT 
scores to his grade point average at graduation. 


Construct validity concerns the degree to which 
measures conform in accordance with theoretical 
expectations. Consequently, a theoretical concept 
must exist for predicting and analyzing the measures. 
Measures are considered to have construct validity if 
they are consistent with theoretical expectations. If 
they are inconsistent with theoretical expectations, 
then it is concluded that the measures lack validity 
for that particular concept (Carmines and Zeller, 
1979). 


Convergent-discriminant validity is the expectation, 
based on a priori or theoretical grounds, that 
measures would relate to some other variable or 


variables. Convergent validity assesses the agreement 
between different methods of measuring the same 
construct. Discriminant validity assesses the 


difference in results when the same method is used to 


measure supposedly different attributes of the same 
object. (Brewer and Collins, 1981) 


To establish the validity of measurements, social 
science researchers verify that the measures correlate with 
at least one of these types of validity. High correlation 
to a specific type of validity allows the assignment of that 
particular type of validity to the measurements. Low 
correlation to any of the above types of validity fails to 
establish the validity of the measurements. Expert 
judgement or empirical procedures serve as the basis for 
establishing correlation, based upon the type of validity 
claimed. Cronbach’s coefficient alpha, Pearson’s product- 
moment correlations, and Spearman’s rank correlations are 
examples of statistical calculations typically used when 
establishing correlation through empirical procedures. 

F. INTERNAL AND EXTERNAL VALIDITY 

The assignment of a type of validity to a given set of 
measurements, by virtue of the existence of a high degree of 
correlation to an established type of validity, does not 
resolve the issue of whether the experimental design 
accurately measures the phenomena of interest. Further, it 
doesn’t establish whether the study findings can be 
legitimately extended beyond the sample. These issues are 
directly addressed as the concepts of internal and external 
validity. The existence of internal and external validity 


establishes the meaningfulness of an experimental design far 
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more than does the assignment of a type of validity to a 
given measurement. 

Campbell and Stanley (1966) define internal validity as 
dealing with the issue of whether the measurements are due 
to the effect of a particular independent variable or 
causally related to the dependent variable of interest. It 
asks whether a difference exists at all in any given 
comparison or if an observed difference is an artifact of 
the observational process. Internal validity provides 
assurances that the measurements from a research design 
result from its identified variables. When the measurements 
are causally related to the dependent variable or other 
factors for which no accounting is made in the research 
design, the internal validity of the measurements becomes 
confounded. 

Campbell and Stanley (1966) define external validity as 
the degree to which findings can be generalized to larger 
populations or settings. One uses external validity to 
project the findings from a experiment beyond its sample, 
onto some larger population of interest. 

G. IMPORTANCE OF INTERNAL VALIDITY 

Internal validity is more important than external 
validity. Measurements from experimental designs that do 
not have internal validity are considered unreliable and 
unrepresentative of the phenomena of interest. 


Consequently, it is impossible to extend a study’s findings 
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beyond the sample population. Internal and external 
validity are interdependent. Without internal validity, an 
experimental design can not have external validity. The 
existence of internal validity does not insure that the 
experimental design will possess external validity. 
H. FACTORS THREATENING INTERNAL VALIDITY 
Campbell and Stanley (1966) identify 12 factors 

jeopardizing the internal or external validity of 
experimental designs. Relevant to internal validity, they 
present eight factors, which if not controlled in the 
research design, might result in effects that obscure the 
effect of the experimental stimulus. Some of these factors 
threatening internal validity deal with spurious conclusions 
based upon observations made before and after the effect of 
an intervening experimental variable. Others deal with 
erroneous conclusions based upon supposed differences 
between comparison groups. The following factors 
potentially threaten the internal validity of experimental 
designs: 

l. History, the external events occurring between the 

pretest measurements and post-test measurements in 
addition to the experimental variable. 


2. Maturation, refers to the process of change within the 
respondents themselves as a function of the passage of 


time. 

3. Testing, the effects resulting from having taken 
earlier tests, commonly referred to as re-test 
sophistication. 

4. Instrumentation, the changes occurring in the 


calibration of the measuring instruments, observers, 


12 


or scorers produces changes in the observed 
measurements. 


5. Statistical regression, where groups have been 
selected on the basis of their extreme scores. 
Measures of such groups tend to regress towards the 
mean on subsequent observations. 


6. Selection biases, is the differential selection of 
respondents for the sample population. 


7. Experimental mortality, is the non-random drop-out 
of participants from the experiment. 


8. Selection-maturation interaction, in multi-group 
designs, their interaction is confounded with the 
effect of the experimental variable. 

I. FACTORS THREATENING EXTERNAL VALIDITY 

Campbell and Stanley (1966) presents four factors that 
can jeopardize external validity or representativeness. If 
not controlled in the research design, these factors might 
result in the inappropriate generalization of study results 
to larger populations. These factors are (1) the reactive 
or interactive effect of testing, (2) the interaction 
effects of selection biases and the experimental variable, 
(3) reactive effects of experimental arrangements, and (4) 
multiple-treatment inference. 

Bernstein et al. (1975) elaborated on the works of 
Campbell and Stanley (1966) and Bracht and Glass (1968), 
regarding factors that threaten external validity of 
research findings. They have grouped these factors into the 
following categories: 

1. Selection effects result from infeasibility of drawing 
a probability sample from the target population. 
Instead the researcher must sometimes use a biased 


sample. The types of such selection biases, in order 
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of the degree to which they threaten external 
validity, are (1) self-selection by the respondents 
into treatment and control groups, (2) selection by 
excellence, (3) selection by expedience, and 

(4) non-random posttreatment matching. 


Measurement effects includes the effects of 
unreliability and invalidity of measurement, the 
reactivity of some measures, and interactions 
between measurement and other variables. 


Confounded treatment effects may arise when (1) a lack 
of standard treatments are applied across sampling 
elements, (2) multiple treatments are applied across 
sampling elements, or (3) individual level variables 
(attributes) of the sampling elements interact with 
the treatments. 


Situational effects result from the conduct of the 
experiment in a particular context that is not 
representative of all contexts to which the researcher 
may wish to extend a generalization. 


Effects due to differential mortality result from 
differential subject and program losses during 

social experiments. When the losses are different for 
the treatment and control groups, and these losses are 
not random, then external validity is threatened. 


DATA COLLECTION TECHNIQUES 


Galtung (1967) states that all social science data are 


obtained in either formal (laboratory) or informal (field) 


settings and involve either verbal (oral and written) or 


nonverbal acts or responses. The combination of the two 


settings for data collection and two ways responses are 


exhibited have led to the development of these commonly used 


data collection techniques: 


1: 


Le 


Observations of behavior 

Experiments 

Reports by others 

Analyses of existing records and documents 
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5. Interviews 
6. Questionnaires. 
Each data collection technique has its own characteristics 
and resulting impacts upon validity. 
1. Observations of Behavior 
There are primarily three forms of collecting 


observations of behavior: 


1. Simple observation 
2. Contrived observation 
3. Participant observation. 


Simple observation of behavior involves an observer who has 
no control over the behavior or situation in question, and 
plays an unobserved, passive and unobtrusive role in the 
research situation (Webb et al., 1966). Denzin (1970) 
identifies five types of simple observations: exterior body 
and physical signs, expressive movement, physical location 
analysis, observation of language, and time sampling 
analysis. 

Denzin (1970) identifies sources of invalidity with 
simple observation. Populations change over time and 
location, making it very difficult to gather representative 
samples. Small samples of much larger populations may 
result in quite tentative generalizations. Observers or 
measuring devices make errors. Some types of behavior occur 
usually under conditions that are very difficult to observe, 


for example covert or illicit acts. Finally, unless prior 


ÉS 


investigations are conducted, it is very difficult to 
position observers in settings where critical behavior 
occurs at a high frequencies. 

Contrived observation substitutes the human record- 
keeping observers present in simple observation with 
hardware devices (Webb et al., 1966). Hardware devices are 
substituted for humans as means of avoiding human instrument 
error and supplementing control of the experiment. It is an 
unobtrusive form of data collection, employing devices such 
as hidden video cameras, mirrors, and audio recorders. 
Measurements obtained through the use of contrived 
observations share the same sources of threats to validity 
aS measurements obtained through the use of simple 
observation. 

Participant observation of behavior is a field 
strategy that simultaneously combines document analysis, 
respondent and informant interviewing, direct participation 
and observation, and introspection (Denzin, 1970). The 
observer can assume one of four roles. 

1. Complete participant role is one in which the observer 
is wholly concealed and he tries to become a full 
member of the investigated group. 


2. Participant as observer role is when the investigated 
group is aware of the identity and intentions of the 


observer. 

3. Observer as participant role is one in which 
investigations usually involve only one visit or 
interview with the respondent. The interaction is 


typically brief and highly formalized, using 
questionnaires or interviews. No attempt is made by 
the observer to establish a sense of a lasting 
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relationship with the respondent--the epitome of the 
stranger encounter. 


The complete observer role involves complete removal 
of the observer from the interaction. The complete 
observer role is an unobtrusive method of collecting 
data, while the other roles are obtrusive. 


Denzin (1970) identifies factors regarding 


participant observation that pose threats to internal 


validity. 


thse, 


Historical factors are events that either occurred 
before observations were made or during the period 
between the first and last observations. 


Subject maturation factors reflect the changes that 
occur in respondents, resulting from their 
relationships with observers. 


Subject bias factors are the nonhomogeneous 
characteristics of the observed subjects. 


Subject mortality factors reflect the number of 
subjects who drop-out of or are dropped from the 
experiment. 


Reactive effects of the observer factors are effects 
that are created through the introduction of the 
observer. 


Observer changes factors are effects that are 
attributed to changes occurring in observers. 


Situation factors are effects caused by peculiar 
aspects of the situation in which the observations 
were conducted. 


The degree to which the sample selected for observation 


actually represents larger populations to which study 


findings are generalized poses concerns of external 


validity. 
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2. Experiments 
Campbell and Stanley (1966) define experiments as 
that portion of research in which variables are manipulated 
in controlled settings and their effects upon other 
variables observed. Denzin (1970) states, 
"In its generic form the experiment represents a 
situation in which the investigator controls some 
variables while he manipulates the effects on others. 
This permits him to observe the effects of the 
manipulated variables upon the dependent variable ina 
situation where the effect of other relevant factors is 
believed to have been removed, typically by 
randomization." 
Experiments are an obtrusive method of data collection. 


Figure 2 represents the characteristics of a generic 


experiment. 
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Figure 2. Pretest and Post-test One Control Group Design 





As discussed in Chapter II, Sections H and I, 
Campbell and Stanley (1966) identified 12 factors 
jeopardizing the internal or external validity of 
experimental designs. Denzin (1970) argues that the 
experiment is the model for causal inference from which all 
other inferential strategies are either derived or to which 


they are compared. Despite the formidable number of threats 
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to the validity of experiments, they provide one of the best 
methods to answer questions of causal inference. The use of 
experimental methods conveys scientific meaning to a study’s 
findings. 
3. Reports by Others 

Researchers often use informants to secure reports 
from others. LeVine (1981) expands on previous works of 
Campbell to define an informant as normally a member of a 
group who draws and communicates generalizations from the 
particularities of his social environment to a researcher. 
The informant generally possesses superior inductive or 
verbal skills or has experienced exposure to the phenomena 
in which the researcher is interested. Denzin (1970) 
further defines the role of the informant as an unobtrusive 
de facto observer for the researcher, an intimate, a 
confidant, and a friend. The presence of this kind of 
binding relationship between the informant and researcher 
distinguishes an informant from a respondent. LeVine (1981) 
identifies the informant’s biases as this method’s primary 
threat to validity. Informants view their environment from 
a particular vantage point. This leads inevitably to biased 
perspective, which poises threats to the internal and 
external validity of findings derived from measurements 


obtained through informants. 


ye 


4. Analyses of Existing Records and Documents 

Webb et al., (1966) state analyses of existing 
records and documents are an unobtrusive method of 
collecting data. Researchers access archives containing 
either vocational or scholarly documents and records to 
obtain relevant data. Computer usage files are an example 
of archival records. 

The selective deposit and selective survival of 
information in archives can introduce a significant degree 
of bias, thereby threatening both internal and external 
validity. Selective survival occurs when existing records 
and documents are not a representative sampling of the 
studied population. Selective deposit occurs when records 
and documents are generated on a non-random basis. 

5. Interviews and Questionnaires 

An interview is a form of survey study that involves 
"a face to face verbal interchange in which one person, the 
interviewer, attempts to elicit information or expressions 
of opinions or belief from another person (Maccoby and 
Maccoby, 1954)." Interviews are an obtrusive method of data 
collection. The basis of all interviews is the questions 
that are asked. Denzin (1970) comments on interviews, "The 
questionnaire or interview must serve two broad purposes: 
it must translate research objectives into specific 
questions, the answers to which will provide data necessary 


for hypothesis testing, and it must assist the interviewer 
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in motivating the respondent so that the necessary 
information is given." An interview is conducted in one of 
three forms: 

1. Schedule standardized 

2. Nonschedule standardized 

3. Nonstandardized. 

The schedule standardized interview involves wording 
and ordering all questions exactly the same for all 
respondents, thus using a uniform measuring instrument. It 
must be assumed that the questions mean the same thing to 
all respondents. Granting that all questions are 
comparable, any variations between respondents are 
attributed to actual differences in the response, not the 
measuring instrument (Denzin, 1970). 

The nonschedule standardized interview involves the 
interviewer working with a standard list of the information 
required from each respondent, but ordering and phrasing the 
questions in such a manner as to best gather the 
information. This allows the interviewer to partially 
tailor the interview to the individual characteristics of 
each respondent. Modification of the interview requires a 
highly skilled interviewer. Granting that the interviewer 
succeeds in asking comparable questions to all respondents 
during such nonschedule standardized interviews, any 
differences are attributed to actual variances in response, 


not the measuring instrument (Denzin, 1970). 
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The nonstandardized interview does not use a pre- 
specified set of questions nor are questions asked in a 
specific order. This allows the interviewer complete 
freedom to explore any issues that arise during the course 
of the interview. It is clearly recognized that respondents 
are not presented with a comparable set of questions 
(Denzin, 1970). 

A questionnaire is another form of survey study that 
is quite similar to a scheduled standardized interview. 
Questionnaires, like an interview, attempt to elicit 
information or expressions of opinions or belief from 
another person or persons. They differ from the interview 
in that the interchange is not verbal. The interchange most 
commonly occurs in the form of written questions and 
answers. Questionnaires are an obtrusive method of data 
collection. 

Denzin (1970) declares that all types of interviews 
and questionnaires share common deficiencies. First, 
researchers relying on interviews and questionnaires can 
experience difficulties penetrating the respondent’s group 
language and mechanisms of symbolism. When failure is 
experienced, they often ask questions concerning what is on 
the mind of the researcher, rather than what is on the mind 
of the respondent. 

Second, errors arise from respondents not always 


interchanging honestly with the interviewer during the 
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conduct of an interview or completion of a questionnaire. A 
variation of this deficiency can occur when questionnaires 
are mailed to targeted respondents. Persons other than the 
targeted respondent can complete mailed questionnaires, 
assuming the identity of targeted respondents. Researchers 
typically assume that questionnaires are completed by the 
targeted respondent, since they possess little means of 
knowing who actually completes questionnaires. 

Third, respondents occupy different positions within 
their own groups, and hence have their own interpretations 
and even distortions of what are the actual group’s values. 
Each of these commonly shared deficiencies potentially 
threaten the internal and external validity of measurements 
when using the interview as a data collection technique. 

6. Physical Traces 

Webb, et. al. (1966) discussed an unobtrusive data 
collection technique that is based on the study of physical 
traces. Researchers analyze physical traces and signs that 
were left behind by a population. The producers of the 
physical traces and signs are unaware of their future use by 
researchers. There are two basic types of physical traces: 
erosion measures and accretion measures. 

Erosion measures are remains of some population’s 
activities that have selectively worn certain objects. The 
path worn on a tile floor is an example of an erosion 


measure. Erosion measures are susceptible to biases 
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resulting from the physical composition of materials in 
which the traces are recorded. 

Accretion measures are the recording of population 
deposits over time. The cigarette butts deposited in an ash 
tray are an example of an accretion measure. Accretion 
measures are susceptible to biases resulting from the 
possibility that materials have selectively survived or been 
selectively deposited. Selective survival occurs when only 
a non-representative sampling of the population exists. 
Selective deposit occurs when physical traces are created on 


a non-random basis. 
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III. METHODOLOGY 


A. SELECTING STUDIES FOR REVIEW 
1. Choosing Sources from which to Obtain Studies 

Journals in information systems and related fields, 
published in English between 1970 and 1990, were the source 
from which studies of information systems effectiveness were 
selected for review. These journals are accepted as 
appropriate professional reading material and publishing 
outlets for information systems academics (Davis, 1980; 
Hamilton and Ives, 1983; Vogel and Wetherbe, 1984; Alavi et 
al., 1989). The journals surveyed were: 


1. Academy of Management Journal 


2. Accounting Review 

3. ACM Computing Surveys 

4. Administrative Science Quarterly 
5. Communications of the ACM 

6. Data Base 


7. Datamation 

8. Decision Sciences 

9. EDP Analyzer 

10. Harvard Business Review 
11. IEEE Computer 

ES. Information and Management 


3 Interfaces 
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14. Journal of Management Information Systems 

15. Journal of Systems Management 

16. Management Information Systems Quarterly 

17. Management Science 

18. Omega 

19. Sloan Management Review 

2. Identifying Specific Studies for Review 
a. On-Line Bibliographic Search 
An on-line search in January, 1989 reported 128 

items using the string of key words: (Productivity or 
Effectiveness) and (Measures or Measurement or Assess or 
Assessment) and Information Systems. Out of these, 18 
articles appeared, on the face of it, to report empirical 
assessments of information systems effectiveness as 
determined by a reading of their titles. None of them, 
despite having been published in information systems 
journals (e.g. MIS Quarterly, Journal of MIS, Information 
and Management, Data Base), are among the 37 empirical 
studies of information systems effectiveness discussed here. 
Some focused on the development of survey measurement 
instruments, a category that was, as it is explained below, 
excluded from this analysis. Other articles were from trade 
and industry publications aimed at MIS practitioners and 
dealt with advice on effectiveness measurement. They did 


not report empirical studies of system effectiveness. 
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b. Manual Bibliographic Search 

It was suspected that the empirical literature 
on information systems effectiveness was more extensive than 
the handful of references revealed by the on-line search. A 
manual search of the same journals was conducted. Every 
issue of all the journals, dating back to 1970, were 
searched. The manual searching brought forth the difficulty 
with the on-line search; deficient key words. Authors, for 
example, used key words such as "manufacturing and 
engineering systems, return on investment, utilization 
process, human information processing, information system 
design, system evaluation, information system investment, 
cost/benefit calculations, management reporting, management 
issues, accuracy, content, information characteristics and 
users evaluation." Such key words were, in retrospect, 
quite appropriate for describing various aspects evaluating 
information systems effectiveness. They simply were not the 
ones that had been imagined to be relevant to the on-line 
search. No single search could have accommodated so many 
key words. While a sequence of searches could have employed 
a large number of key words, the limited on-line searching 
resources available for this study did not make that 
alternative feasible. 

The manual effort produced 118 articles that 
were identified (by title, key words and abstracts) as 


dealing with the evaluation of information systems 
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effectiveness. The reference lists in these articles, in 
turn, identified another 31 articles that had been 
overlooked because of title, bringing the total number of 
items in the literature base to 149. Most of these articles 
did not report empirical studies. 
c. Selection of Specific Studies 

Excluded immediately were non-empirical analyses 
of the assessment of information systems (Chervany and 
Dickson, 1970; Coleman and Riley, 1972; Mason and Mitroff, 
1973; Carlson, 1974; Soergel, 1976; Strassmann, 1976; Tharp 
and Taggart, 1976; Radecki, 1976; Dickson et al., 1977; 
Arnovick and Gee, 1978; King and Schrems, 1978; Ginzberg, 
1978; Mason, 1978; Kleijnen, 1979; Worthley, 1980; Ginzberg, 
1980; Hamilton and Ives, 1982; Borko, 1983; Keim and Janaro, 
1982; Markus and Robey, 1983; Mason, 1984; Calista, 1985; 
Bearman et al., 1985; Gremillion and Pyburn, 1985; Lay, 
1985; Maggiolini, 1986; Martin and Trumbley, 1986; Rouse, 
1987; Marsden and Pingry, 1988; Trice and Treacy, 1988). 

Also excluded from this review were empirically- 
based articles that do not deal directly with a study of 
information systems effectiveness but rather were concerned 
with the validation of attitudinal questionnaires and other 
instruments (Larcker and Lessig, 1980; Mick et al., 1980; 
Chandler, 1982; Bailey and Pearson, 1983; Srinivasan, 1985; 
Dominick, 1987; Hill et al., 1987; Miller and Doyle, 1987; 


Baroudi and Orlikowski, 1988; Money et al., 1988; Montezema, 
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1988). While these works involve the development of 
research techniques that contribute to how we know what we 
know about information systems effectiveness, they were left 
out because they were not primarily engaged in studies that 
had a potential for developing theory about behavior related 
to information systems effectiveness. 

A miscellany of pieces that seemed to have the 
titles sought but actually dealt with other issues were also 
removed from the original 149 articles. Those addressing 
evaluations of the technical performance of hardware (e.g. 
computer processor performance in millions of instructions 
per second) were not included. Similarly, articles dealing 
with efficiency in software development activities (e.g. 
programmer productivity in terms of lines of code written 
per day) were excluded. 

Finally, my search efforts boiled down to the 37 
reports of empirical studies of the effectiveness of 
information systems in its various formulations. A listing 
of these reports is provided at Appendix A. Surely, for all 
of this, some studies may have been missed, possibly some 
important studies. For that I apologize in advance to those 
researchers who might have been overlooked. It is my 
contention that even though every last empirical study of 
information systems effectiveness may not have been included 
in this research, I do believe that those included are 


representative of the field. 
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B. REVIEWING THE STUDIES 

The 37 empirical studies (listed in Appendix A) were 
individually reviewed to identify data collection 
instruments and discussions of validity issues. Researchers 
often used multiple data collection instruments in the 
conduct of their studies, with each instrument contributing 
only a portion of the measurements upon which the 
researchers’ findings were based. This use of multiple data 
collection instruments necessitated the development of a 
methodology to identify and separately analyze each 
instrument used in the 37 studies. 

Each study was carefully read to identify all data 
collection instruments contributing to the findings of the 
study. Each instrument was further analyzed to record 
information concerning its design, purported focus of 
measurement, administration, and validity. The analysis 
process for each study essentially consisted of the 
following: 

1. Identification and categorization of all data 
collection instruments used by the information systems 
researchers for the given study. 

2. Categorization of each identified data collection 
instrument according to what the information systems 


researchers purported it to measure. 


3. Examination of the way in which each identified data 
collection instrument were administered. 


4. Identification of how the information systems 


researchers dealt with the validity of each identified 
data collection instrument. 
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The primary manner in which data were collected during 
the analysis of each data collection instrument was through 
the careful reading of the 37 studies. Pertinent 
information was identified and extracted from the studies, 
clearly requiring the exercise of personal judgment. 
Consequently, all findings resulting from this research are 
ultimately based upon face validity. The remainder of this 
section provides details on how the analysis was conducted 
and recorded. 

1. Identification and Categorization of Instruments 

The initial goal of the review was to identify data 
collection instruments used by the information systems 
researchers to measure aspects of information systems. 
Instruments that were used solely in the design of other 
measuring instruments, for example preliminary interviews 
conducted to design questions appearing later ina 
questionnaire, were not considered of interest to this 
research. 

Each data collection instrument was classified by 
the data collection technique it primarily employed. 
Instruments were classified as questionnaires, interviews, 
archival records, and observations. 

Each data collection instrument was further analyzed 
to determine specific design characteristics of the 
technique. Data were gathered on the specific design of 


each questionnaire, interview, collection of archival 
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records, or set of observations used by the researchers. 
The type of scaling and original source of each 
questionnaires was recorded. The amount of schedule and 
structure existing in each interview was recorded. The 
source and types of records used for each collection of 
archival data was recorded. The manner in which 
observations were conducted was recorded. 

2. Categorization of Instruments by Measures 

It is of interest to determine what information 

systems researchers purported to measure with data 
collection instruments. Each instrument is purported by its 
designer to provide one or more general types of 
measurements relating to information systems. This study 
categorizes instruments according to the type or types of 
measurements it purportedly records. The instruments used 
by researchers were categorized as purportedly measures of 
the following: 


1. Attitude or perception 


2. Behavior 
3. System characteristics 
4. Situational characteristics. 


Each instrument used in the 37 studies was analyzed to 
determine the category or categories of measures that its 
designers claimed it provided. 

Measures of attitude or perception are provided by 


respondents when solicited by researchers. This usually 
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involves respondents providing opinions or testimony on some 
aspect of an information system. Consequently, the 
researchers are indirectly measuring the information system 
through the observations provided by others. Measures of 
attitude or perception typically involve aspects of 
information systems such as user involvement, effectiveness 
of systems, and satisfaction with systems. 

Measures of behavior record the occurrence of 
behavior related to information systems. The behavior 
usually takes the form of interactions between users and 
information systems or actions taken as a result of outputs 
provided by information systems. Measures of behavior 
usually involve the researchers directly observing or 
deducing specific behavior relative to information systems 
or their users. Observation and archival records are often 
used to collect measures of behavior. Data recorded through 
the review of computer inquiry logs or account records 
regarding system usage is an example of measures of 
behavior. 

Measures of system characteristics are data which 
purport to directly represent some aspects of information 
systems. Examples are cost of systems, the effectiveness of 
systems, the impact of systems, the success of systems, or 
the satisfaction of users. Measures of system 


characteristics differ from measure of attitude or behavior 
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in that they are purported to directly measure aspects of 
information systems. 

Measures of situational characteristics provide data 
concerning the environment within which the system operates. 
Measures of situational characteristics are concerned with 
describing components of the environment that essentially 
impact upon information systems. Measures of situational 
characteristics differ from the previously discussed types 
of measures in that they do not record data on the impact of 
the studied information systems; however they do provide 
insight into the conditions surrounding information systems. 
Examples of components of the environment that are measured 
by researchers are organizational structure, historical 
performance of business, personnel profiles, and industry 
trends. 

3. Examination of the Way Instruments are Administered 

The way in which data collection instruments are 
administered is an essential part of the design of an 
experiment. It involves the selection of potential 
respondents, distribution of the data collection 
instruments, and follow-up actions taken to encourage 
response and account for non-response. The manner in which 
researchers administer instruments can fundamentally affect 
the validity of data. 

The goal of this examination is to provide 


information on how researchers are administering data 
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collection instruments. Analysis of the sampling process 
used by researchers may provide information regarding the 
validity of the data they collect. Each identified 
instrument was examined to determine the following: 

1. Who responded to each instrument. 


2. The size of the sample taken with each 
instrument. 


3. The response rate achieved for each instrument, with 
the exception of archival records. 


4. How each instrument was distributed. 


5. Whether follow-up actions were taken to improve the 
participant response rate for each instrument. 


6. Whether differences between respondents and non- 
respondents to each instrument were identified by the 
information systems researchers. 

4. Identification of how Researchers dealt with 

Validity Issues 

The primary focus of this study is to explore 
validity issues stemming from measurements of information 
systems. Identifying how information systems researchers 
discussed, claimed, and substantiated the validity of data 
collected during their studies provides a basis for 
assessing the role validity issues play in information 
systems research. The following information was recorded 
for each data collection instrument in order to establish 
how researchers dealt with validity issues: 

1. How internal validity was addressed. 


2. How external validity was addressed. 
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3. The types of validity claimed or implied. 
4. The source of the validity claimed or implied. 

All discussion regarding the validity of 
measurements derived from a given instruments were recorded. 
Secondary sources were examined and noted when cited ina 
discussions. All recorded discussions were then determined 
to address internal and or external validity issues, and 
classified as such. The classified discussions were then 
further summarized for ease of understanding. When no 
discussions of internal and or external validity were 
presented, it was so noted as not discussed. Further 
analysis was not required for instruments without 
discussions of internal and external validity. 

Data collection instruments having discussions of 
internal and or external validity were further analyzed. 

The discussions were determined to claim, imply, or deny the 
validity of measurements. The types of validity claimed or 
implied were noted, as well as denials of validity. Whether 
validity was claimed or implied was also noted. 

When the validity of measurements was claimed or 
implied for a given data collection instrument, the source 
of the validity was noted. It was also noted when 
information systems researchers claimed or implied validity 
but failed to cite a source. 

Discussions that collectively address the validity 


of measurements derived using multiple data collection 


36 


instruments in a given study were recorded. These 
discussions were further analyzed as if they represented a 
Single data collection instrument. The results of such 
analyses are identified as "collectively", when such 
discussion were present in a study. 
C. RECORDING THE DATA 

Data were extracted from the 37 studies using a separate 
summary sheet for each study. (Appendix B) In order to 
consolidate the summary sheets into a more understandable 
format, a matrix was created to present the data as detailed 
in the previous section. (Appendix C) Both the summary 
sheets and the matrix are used as sources for the findings 


of this study. 
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IV. FINDINGS 


Appendix B contains a detailed analysis of the 37 
analyzed studies. Appendix C is a consolidation of data 
presented in Appendix B. Both of these appendices were used 
as sources for the findings that follow. 

A. INSTRUMENTS USED BY RESEARCHERS 
1. Summary of Instruments 
The number and types of data collection instruments 
used by information systems researchers during the 37 
studies are shown in Table 1. 
TABLE 1 


DATA COLLECTION INSTRUMENTS USED 


INSTRUMENT # STUDIES USING # TIMES USED 


— 






| Archival 


Interview 


me 

















Observation 


The column labeled # STUDIES USING contains data 
regarding the number of studies that used specified types of 
instruments. The column labeled # TIMES USED contains data 
regarding the number of times each type of instrument was 


used during the 37 studies. The number of instruments used 
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exceeded the number of studies because some researchers used 
multiple data collection instruments during the conduct of a 
Single study. 

Table 1 reveals that questionnaires were used in 
over three-quarters of the studies. Twenty-nine of 37 
studies included questionnaires. Data collection which 
involved archival records, interviews, or observations was 
less frequent. When these latter data collection techniques 
were used, they were often used in conjunction with 
questionnaires. 

The 37 studies used 65 individual data collection 
techniques. Over half of the data collection techniques 
used were questionnaires, with 36 of the 65 techniques 
having been classified as questionnaires. 

2. Use of Multiple Data Collection Instruments 

Table 2 illustrates the use of multiple data 
collection instruments by information systems researchers. 
Use of multiple data collection instruments denotes the 
incorporation of more than one data collection instrument in 
a given study. For example, Lucas (1974) used two 
questionnaires and Colton (1972) used a questionnaire and an 
interview, thus each used multiple data collection 
instruments. Table 2 shows the number of studies that used 


one, two, three, or four data collection techniques. 
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TABLE 2 


USE OF MULTIPLE DATA COLLECTION INSTRUMENTS 










NUMBER OF INSTRUMENTS USED 
One Two Three Four 


NUMBER OF STUDIES 


Table 2 shows that over half of the researchers used 
only one instrument in their research designs. A further 
analysis of the 19 one-technique studies revealed that they 
used ten questionnaires, six sets of archival records, two 
collections of observations, and one interview. 

3. Characteristics of Questionnaires 

Details concerning design characteristics were 
provided for 31 of the 36 questionnaires used in the 37 
studies. Even when discussions were presented, they rarely 
extended beyond the identification of the types of scales 
used to score instruments. Few studies clearly identified 
the origin or source of questionnaires used. Discussions 
were insufficient to develop findings about the structure of 
questionnaires. | 

Questionnaires used by researchers were largely 
based upon ordinal scales. Of the 31 questionnaires that 
discussed their characteristics, 30 were based upon ordinal 
scales. Five, seven, and ten point ordinal scales were 
typically used in the design of the questionnaires. Five 
studies contained discussions specifically stating that 
their corresponding questionnaires were Likert-type; Franz 
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and Robey (1986), Aldag and Power (1986), Franz et al. 
(1986), Gallupe and DeSanctis (1988), Tait and Vessey 
(1988) . 

The origin of the questionnaires were discussed for 
a third of the questionnaires used. The absence of 
discussions left uncertain whether the questionnaires were 
of original design or were developed during previous 
studies. While none of the questionnaires were explicitly 
identified as having been of original design, however, it is 
possible that some were. Eleven of the 36 questionnaires 
were identified as developed during previous studies. These 
questionnaires were attributed to one or more of the 
following: Smith et al. (1969), Lucas (1973), Schultz and 
Slevin (1975), Wolek (1975), Elizur and Guttman (1976), 
Sims et al. (1976), Ein-Dor and Segev (1978), Keen (1981), 
McFarlan (1981), Bailey and Pearson (1983), Ives et al. 
(1983), Baroudi and Orlikowski (1988), and Doll and 
Torkzaheh (1988). 

4. Sources of Archival Records 

Researchers used archival records 15 times in the 37 
studies. They provided discussions identifying the sources 
of the archival records for 13 of the 15 instances in which 
they were used. Over two-thirds, 9 of the 13 discussions, 
identified the archival records as computer files or 


computer generated reports. Insufficient information was 
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provided in the discussions to clearly establish the sources 
of the remaining archival records. 

5. Characteristics of Interviews 

Nine interviews were identified in the 37 studies. 

Little discussion was presented regarding these interviews. 
No determination of structure or schedule could be made for 
six of the nine interviews. Three interviews cited 
combinations of structured, unstructured, scheduled, and 
unscheduled designs. 

6. Characteristics of Observations 

Observations were identified four times as the data 

collection technique in the 37 studies. The researchers 
discussed the manner in which the observations were made 
each time they were identified as used. These discussions 
were used to determine observer roles, as defined by Denzin 
(1970). Three of the observations were collected by 
participant observers. The fourth set was collected by a 
complete observer. 
B. WHAT RESEARCHERS PURPORTED TO MEASURE 

Instruments used in the 37 studies were categorized as 
purported to measure of attitude, behavior, system 
Characteristics, or situational characteristics. Measures 
of attitude typically consisted of stated opinions or 
testimony as provided by respondents. Measures of behavior 
recorded the occurrence of behavior related to information 


systems. Measures of system characteristics are data which 
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are purported to directly represent aspects of information 
systems. Measures of situational characteristics provide 
data concerning the environment within which systems 
operate. Further explanations of these categories of 
measures are provided in Chapter III. 

Table 3 shows what the researchers purported to measure 
in the 37 studies. Some instruments purportedly collected 
more than one category of measures. For example, Swanson 
(1974) used a single questionnaire to purportedly measure 
both attitude and behavior. Consequently, the total number 
of purported measures is greater than the total number of 
instruments used. 

TABLE 3 


PURPORTED MEASURES OF INSTRUMENTS 
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Table 3 shows that the 65 instruments made a combined 





total of 85 purported measures, distributed between four 
categories. Questionnaires were the only instrument 


identified to purportedly measure multiple categories of 
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measures with a single data collection instrument. 
Questionnaires were also the only instrument identified to 
purportedly measure all categories of measures. 
Questionnaires and interviews, when considered together, 
were identified 27 times as purported measures of attitudes 
and 38 times as purported measures of the remaining 
sateen Analysis of archival records were identified to 
purportedly measure all categories except attitudes. 
Observations were identified solely as purported measures of 
systems characteristics. 
C. THE WAY IN WHICH INSTRUMENTS WERE ADMINISTERED 
1. Selection of Sample Populations 

The information systems researchers sampled 
populations from a variety of professions and functional 
specialties. All 37 studies appear to have used samples 
selected for convenience or availability. Randomness was 
introduced into the sampling process only after the sample 
populations were identified. 

2. Identity of Respondents 

A total of 49 questionnaires, interviews, and 
observations were used in the 37 studies. Managers, 
computer personnel, and computer users were identified as 
respondents or observed subjects for 43 of the 49 
instruments. The remaining six instruments used students as 
respondents or observed subjects. King and Rodriguez 


(1978), Aldag and Power (1986), Durand et al. (1987), and 
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Gallupe and DeSanctis (1988) were the studies that used 
students. All other studies relied on respondents or 
observed subjects that used information systems in 
operational environments. 
3. Size of Samples 

Information identifying the size, duration, or 
frequency of sampling was provided for only 38 of the 65 
data collection instruments. When such information was 
provided, it typically consisted of identifying only the 
total number of respondents to given instruments. The total 
number of respondents was provided for 34 instruments, as 
shown in Table 4. The duration of sampling was provided in 
three studies; Swanson (1974), Ball (1980), and Ginzberg 
(1981). The frequency of sampling was provided in one 
study, Englander and Englander (1985). 


TABLE 4 


SAMPLE SIZE 





SAMPLE SIZE 
1 to 50 51 to 100 





101 to 500 


% of INSTRUMENTS 


4. Response Rates 
A total of 40 questionnaires and interviews were 
used in the 37 studies. Researchers identified the response 
rates for only 20 of these instruments. Table 5 shows the 
response rates that were identified for each type of 
instrument. 
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TABLE 5 


DISTRIBUTION OF IDENTIFIED RESPONSE RATES 


TYPE OF 
INSTRUMENT 


Questionnaire 






RESPONSE RATE ACHIEVED | 
0-20%, 21-40%, 41-60%, 61-80%, 81-100%) 


= Se — mem, 
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Further analysis of the response rates reveals that 
the researchers rarely reported rates higher than 95 
percent. Only three of the 21 instruments exceeded a 95 
percent response rate; questionnaires used by Lucas (1975b), 
Robey (1979), and Bruwer (1984). A 100 percent response 
rate was achieved for only the questionnaires used by Robey 
(1979) and Bruwer (1984). Unfortunately, neither study 
discussed how the 100 percent response rate was achieved. 
At the opposite extreme, the findings of two studies were 
based on instruments achieving a response rate of less than 
40 percent: questionnaires used by Snitkin and King (1986) 
and Baroudi et al. (1986). Snitkin and King (1986) achieved 
a response rate of 18 percent and Baroudi et al. (1986) of 
25 percent. 

5. Distribution Methods 

Data were gathered regarding the manner in which 
questionnaires were distributed and interviews conducted. A 
total of 36 questionnaires were used in the 37 studies. The 


means of distribution was clearly identified for 18 of the 
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36 questionnaires. Mail was identified as the means of 
distribution for 15 of the questionnaires. Three of the 
questionnaires were identified as administered in person. 

A total of nine interviews were used in the 37 
studies. The manner in which the interviews were 
administered was identified for six of the nine interviews 
conducted. These six interviews were identified as 
administered in person. No interviews appear to have been 
administered by telephone. 

The information systems researchers provided little 
discussion regarding actions taken to improve response 
rates. A total of 40 instruments used questionnaires and 
interviews, yet only six instances of follow-up actions were 
clearly identified. All six instances involved 
questionnaires. Colton (1972) used a second mailing. Lucas 
(1975a) and (1975b) stated that unspecified follow-up 
actions were taken for each questionnaire. Ginzberg (1981) 
used a follow-up message for his two questionnaires. Tait 
and Vessey (1988) used a telephone call. 

6. Differences between Respondents and Non-Respondents 

The 40 data collection instruments that used 
questionnaires or interviews were examined to determine 
whether researchers discussed any differences between 
respondents and non-respondents. Discussion of differences 


were presented for ten of these instruments, all of which 
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were questionnaires. When a discussion of differences was 
presented, it typically provided sparse information. 

Ginzberg (1981) presented data, based on an analysis 
of both groups, suggesting that no significant, nor 
sizeable, difference existed between respondents and non- 
respondents to his three questionnaires. Cron and Sobol 
(1983) specifically identified major non-respondents and 
discussed differences. Robey (1979) and Bruwer (1984) 
claimed response rates of 100 percent, thus no discussion of 
non-respondents was necessary. Snitkin and King (1986) 
stated they were prevented from performing an analysis of 
differences. Baroudi et al. (1986) linked non-respondents 
to the use of old mailing lists. Franz et al. (1986) 
performed an analysis of the attitudes of dropouts from his 
experiment, which was used to suggest differences between 
respondents and non-respondents to his two questionnaires. 
D. HANDLING OF VALIDITY ISSUES 

1. Discussions of Internal Validity 

Discussions regarding the internal validity of 
individual instruments were presented for only 19 of the 65 
instruments. Table 6 identifies, by type of instruments 


used, the presentation of such discussions. 
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TABLE 6 


DISCUSSIONS REGARDING INTERNAL VALIDITY 


TYPE of TIMES TIMES 
A DISCUSSED NOT DISCUSSED 
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Analysis of the 19 discussions reveals that the 
researchers addressed internal validity issues in a variety 
of ways. References detailing ways in which specific 
studies addressed internal validity are provided in 
subsequent paragraphs. Most frequently the discussions 
identified secondary sources as the original developers of 
their instruments. Some discussions clearly cautioned 
readers that the internal validity of data collection 
instruments had not been demonstrated. Least common were 
discussions that asserted that internal validity had been 
demonstrated. 

Eleven of the discussions cited secondary sources as 
a means of possibly substantiating the internal validity of 
individual data collection instruments; Lucas (1974), Lucas 
(1975b), King and Rodriguez (1978), Robey and Zeller (1978), 


Lucas (1978), Robey (1979), Baroudi et al. (1986), Franz et 
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al. (1986), Tait and Vessey (1988), and Cronan and Douglas 
(1990). Appendix B identifies the specific source(s) cited 
by each of these studies. The citation of secondary sources 
implies that the individual instruments were validated in 
previous studies. One can expect cited secondary sources to 
provide information about the internal validity of 
individual instruments. 

When the secondary sources cited by Lucas (1974), 
Lucas (1975b), King and Rodriguez (1978), Robey and Zeller 
(1978), Robey (1979), and Franz et al. (1986)* were 
examined, little discussion of internal validity was found. 
The cited secondary sources that provided insufficient 
discussion of internal validity were Lucas (1973), Schultz 
and Slevin (1975), Wolek (1975), Ein-Dor and Segev (1978), 
Keen (1981), and McFarlan (1981). 

Secondary sources cited by Lucas (1978), Baroudi et 
al. (1986), Franz et al. (1986)*, Tait and Vessey (1988), 
and Cronan and Douglas (1990) directly discussed the 
internal validity of individual instruments. The cited 
secondary sources that provided adequate discussion of 
internal validity were Smith et al. (1969), Elizur and 


Guttman (1976), Sims et al. (1976), Bailey and Pearson 


“Reference to secondary source of Sims, Szilagyi, 
Keller (1976) provided discussion of validity issues. 


“Reference to secondary source of Schultz and Slevin 
(1975) provided little discussion of validity issues. 
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(1983), Ives et al. (1983), Baroudi and Orlikowski (1988), 
and Doll and Torkzaheh (1988). 

Six discussions acknowledged deficiencies regarding 
the internal validity of individual instruments. Gallagher 
(1974) informed readers that the validity of his 
questionnaire had not been established. Edstrom (1977) 
identified weaknesses in his interview stemming from 
problems associated with human memory, perceptions, and 
selection of respondents. Englander and Englander (1985) 
suggested that variables not measured by their archival 
records may have adversely impacted the results of their 
experiment. Lincoln (1986) addressed the unreliability of 
user interviews. Baroudi et al. (1986) expressed concerns 
over the ordering of experimental variables and the use of 
subjective rather than objective measures. Tait and Vessey 
(1988) acknowledged problems resulting from when measures 
were collected, influences on respondent’s attitudes, the 
use of Likert-type scales, and insufficient validation of 
instruments. 

In four cases, researchers implied or asserted that 
instruments had demonstrated internal validity. Lucas 
(1978) asserted that the quasi-experimental research design 
and statistical techniques used provided increased 
confidence. Mansour and Watson (1980) implied that their 
questionnaire possessed internal validity on the basis of 


its reliability as a predictor. Franz and Robey (1986) 
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stated that they developed and validated their 
questionnaire. They provided details of how this was 
accomplished. Aldag and Power (1986) implied, through 
persuasive discussion, that their questionnaire had 
demonstrated internal validity. 
2. Discussions of External Validity 
Information systems researchers presented 
discussions regarding the external validity of individual 
instruments for only 22 of the 65 instruments. Table 7 
identifies, by type of instrument, the presentation of such 
discussions. 
TABLE 7 


DISCUSSIONS REGARDING EXTERNAL VALIDITY 
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Analysis of the 22 discussions reveals that the 
researchers also addressed external validity issues ina 
variety of different ways. References detailing ways in 
which specific studies addressed internal validity are 


provided in subsequent paragraphs. Secondary sources were 
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referenced as the original designers of individual data 
collection instruments. In other discussions, researchers 
generalized their results without providing a 
justification. Some discussions clearly cautioned readers 
that the external validity of instruments had not been 
demonstrated. 

References to secondary sources were considered an 
attempt to substantiate the external validity of individual 
data collection instruments. Eleven studies cited secondary 
sources; Lucas (1974), Lucas (1975b), King and Rodriguez 
(1978), Robey and Zeller (1978), Lucas (1978), Robey (1979), 
Baroudi et al. (1986), Franz et al. (1986), Tait and Vessey 
(1988), and Cronan and Douglas (1990). Appendix B 
identifies the specific source(s) cited by each of these 
studies. As noted earlier, the citation of secondary 
sources implies that the individual instruments were 
validated in previous studies. Cited secondary sources are 
expected to provide readers with information regarding the 
external validity of individual instruments. 

When the secondary sources cited by Lucas (1974), 
Lucas (1975b), King and Rodriguez (1978), Robey and Zeller 
(1978), Robey (1979), and Franz et al. (1986)? were 
examined, little discussion of external validity was found. 


The cited secondary sources that provided insufficient 


“Reference to secondary source of Sims, Szilagyi, 
Keller (1976) provided discussion of validity issues. 
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discussion of external validity were Lucas (1973), Schultz 
and Slevin (1975), Wolek (1975), Ein-Dor and Segev (1978), 
Keen (1981), and McFarlan (1981). 

Secondary sources cited by Lucas (1978), Baroudi et 
al. (1986), Franz et al. (1986)‘, Tait and Vessey (1988), 
and Cronan and Douglas (1990) directly discussed the 
external validity of individual instruments. The cited 
secondary sources that provided adequate discussion of 
external validity were Smith et al. (1969), Elizur and 
Guttman (1976), Sims et al. (1976), Bailey and Pearson 
(1983), Ives et al. (1983), Baroudi and Orlikowski (1988), 
and Doll and Torkzaheh (1988). 

Seven of the discussions about external validity 
involved researchers generalizing the findings of their 
studies. By having generalized findings, these researchers 
were considered to have at least implied that the external 
validity of their individual instruments had been 
demonstrated. Elchesen (1978), Mansour and Watson (1980), 
Altmeyer and Bozeman (1981), Cron and Sobol (1983), Bruwer 
(1984), Decker and Plumlee (1985), and Kapsales (1986) all 
generalized findings resulting from their individual 
instruments. Cron and Sobol (1983) were the only 
researchers that explicitly discussed the external validity 


of their individual instruments. 


“Reference to secondary source of Schultz and Slevin 
(1975) provided little discussion of validity issues. 
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Six of the discussions of external validity 
acknowledged deficiencies. Gallagher (1974) and Franz and 
Robey (1986) stated that the validity of their 
questionnaires had not been established. Tait and Vessey 
(1988) implied that the validity of their questionnaire had 
not been established. Edström (1977) cautioned against 
generalizing the findings from his interviews. Cron and 
Sobol (1983) and Aldag and Power (1986) cautioned that 
further studies involving their questionnaires were 
required. 

Baroudi et al. (1986) provided the only discussion 
of actions taken which allegedly enhanced the abilities to 
generalize their results. Researchers typically either 
ignored external validity issues or generalized findings 
without supporting discussions. 

3. Types of Validity Claimed or Implied 

Researchers provided discussions about either the 
internal or external validity of 25 of the 65 individual 
data collection instruments. Specific types of validity 
were claimed or implied for only 15 of the 25 instruments. 
The remaining ten instruments could not be linked to a 
specific type of validity. 

As previously elaborated, discussions regarding six 
instruments involved only references to secondary sources 
that were found to provide little or no discussions of 


validity issues. Consequently, such discussions were not 
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considered as having claimed or implied a specific type of 
validity for the involved instruments. Other discussions 
were presented asserting that four instruments had not been 
validated, again not identifying a specific type of 
validity. 

An analysis of the discussions which claimed or 
implied a specific type of validity for the remaining 15 
instruments reveals that the researchers rarely explicitly 
claimed validity. Explicit claims of validity were made for 
four instruments. In contrast, they implied specific types 
of validity for ten instruments. 

The discussions that explicitly claimed validity 
involved four of the 37 studies. Discussions identifying a 
specific type of validity were presented for five of the 65 
instruments used. All instruments with such discussions 
were questionnaires. 

Aldag and Power (1986) made claims of internal 
validity for their questionnaire. Baroudi et al. (1986) 
made claims of internal and external validity, specifically 
citing content and construct validity for one of their 
questionnaires. Franz et al. (1986) made claims of internal 
and external validity, specifically citing construct and 
discriminant-convergent validity for one of their 
questionnaires. Cronan and Douglas (1990) also made claims 


of internal and external validity, specifically citing 
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construct, convergent-discriminant, and criterion validity 
for their two questionnaires. 

An analysis of the discussions which addressed 
validity, having excluded references to secondary sources 
that provided little or no information regarding validity 
issues, revealed that researchers implied specific types of 
validity for ten instruments. The internal validity of two 
instruments were implied; individual questionnaires used by 
Lucas (1978) and Franz and Robey (1986). The external 
validity of six instruments were implied; observations 
collected by Elchesen (1978), questionnaires administered by 
Altmeyer and Bozeman (1981), Cron and Sobol (1983), Bruwer 
(1984), and archival records analyzed by Decker and Plumlee 
(1985), and Kapsales (1986). Both the internal and external 
validity of two instruments were implied; a questionnaire 
administered by Mansour and Watson (1980) and observations 
collected by Durand et al. (1987). 

4. Sources of Claimed or Implied Validity 

Five questionnaires claimed a specific type of 
validity; Aldag and Power (1986), Baroudi et al. (1986), 
Franz et al. (1986), and Cronan and Douglas (1990). They 
discussed the analyses of instrument measures as 
Justification for claiming specific types of validity. 

Ten data collection instruments were implied to 
possess a specific type of validity; Elchesen (1978), Lucas 


(1978), Mansour and Watson (1980), Altmeyer and Bozeman 
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(1981), Cron and Sobol (1983), Bruwer (1984), Decker and 
Plumlee (1985), Franz and Robey (1986), Kapsales (1986), 
and Durand et al. (1987). None of the discussions which 
involved these instruments identified the source of the 
validity. 

5. Collectively Addressing Validity Issues 

Information systems researchers used multiple data 
collection instruments in 18 of the 37 studies. Fourteen of 
the studies that involved multiple instruments included 
discussions that addressed validity issues in a collective 
manner. Essentially, the researchers addressed the 
collective validity of the study, rather than the validity 
of individual instruments. 

Data were collected regarding discussions that 
collectively addressed either internal or external validity 
issues. These discussions were used to determine the 
specific types of validity that researchers implied or 
claimed their studies demonstrated. None of the discussions 
identified the sources of implied or claimed validity. 

Analysis of the 14 discussions which collectively 
addressed validity issues reveals researchers asserted the 
validity of six studies had not been established; Swanson 
(1974), Lucas (1975a), Lucas (1975b), Schewe (1976), Neumann 
and Segev (1979), and Robey (1979). A single study, Gallupe 
and DeSanctis (1988), included discussion that implied only 


internal validity had been collectively demonstrated. Three 
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studies included discussions that implied only external 
validity had been collectively demonstrated; Hall and 
Lincoln (1976), Robey and Zeller (1978), and Ginzberg 
(1981). Another two studies, Powers and Dickson (1973) and 
Lucas (1974), claimed that only external validity had been 
collectively demonstrated. Two studies implied that both 
internal and external validity had been collectively 
demonstrated; Franz et al. (1986) and Cronan and Douglas 
(1990). Additionally, Cronan and Douglas (1990) was the 
only study to collectively cite specific types of internal 
validity; construct and convergent validity. 

Only two of the 37 studies included discussions that 
individually and collectively addressed validity issues. 
Franz et al. (1986) and Cronan and Douglas (1990) included 
discussions regarding the validity of individual 
instruments. They also provided discussions regarding the 


collective validity of their studies. 
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V. CONCLUSIONS 


The conclusions that follow are based upon key findings 
presented in the previous chapter. The conclusions are 
ordered in same sequence as the findings. Only summary 
totals are provided in this chapter. References to specific 
instruments and studies which comprise the summary totals 
are located with the corresponding findings, in the previous 
chapter. 
A. DATA COLLECTION INSTRUMENTS 

1. Selection of Instruments 

Questionnaires were the primary data collection 

instrument used by researchers. Questionnaires were used in 
29 of the 37 studies, accounting for 36 of the 65 
instruments. This suggests that the majority of the data 
from which researchers generate findings are collected with 
questionnaires. Their reliance on questionnaires leaves 
researchers dependent upon respondents to report on 
effectiveness of information systems. Denzin (1970) 
discussed threats to validity associated with the use 
questionnaires. The obtrusive nature of questionnaires, 
respondent biases, and human memory errors are likely to 
result in many respondents providing inaccurate reports of 


information systems effectiveness. 
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Researchers relied on analysis of archival records 
in a minority of their studies, representing only 15 of the 
65 instruments. Information systems are, by their very 
nature, generators and repositories of archival records. 
The archival records stored in computer files provide an 
unobtrusive manner in which to collect generally accurate 
Bla regarding information systems. Surprisingly few 
researchers took advantage of archival records as a means of 
studying information systems. 

Few researchers used alternative data collection 
techniques, such as interviews or observations. These 
techniques accounted for only 13 of the 65 instruments. 
Difficulties associated with administering interviews and 
collecting observations may account for their less frequent 
use in the 37 studies. 

Interviews require direct person-to-person 
exchanges, which are usually time consuming and costly to 
conduct. Researchers may have opted to use questionnaires 
rather than interviews, since questionnaire are generally 
easier and cheaper to administer. Interviews were 
identified as only 9 of the 65 instruments. Interviews are 
subject to many of the data accuracy problems associated 
with questionnaires. They are obtrusive, susceptible to 
personal biases, and prone to human memory errors. 

Observations are difficult to collect outside the 


laboratory; consequently they are seldom used to measure 
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operational information systems. Observations were the 
bases for only four of the 65 instruments. Observations 
provide accurate data, yet researchers appear reluctant to 
develop experimental designs that use this data collection 
technique. 
2. Impact upon Triangulation 

Multiple data collection instruments were used in 
fewer than one-half of the studies’ research designs. Most 
researchers were precluded from triangulating measures 
because 19 of the 37 studies involved research designs that 
used only one instrument. A greater use of multiple 
instruments would have provided more opportunities for 
triangulation, enhancing the validity of data collected in 
the studies. 

3. Problems with Designs of Questionnaires 

The researchers demonstrated a reliance on ordinal 
scales, using them in 30 of the 31 questionnaires. Only 
five studies explicitly identified the questionnaires as 
Likert-type. Because the researchers consistently discussed 
the use of five, seven, and ten point scales, most of the 
questionnaires were presumedly Likert-type. Galletta and 
Lederer (1987) cautioned that numerous problems are 
associated with Likert-type scales. These problems include 
scale units and origins (anchoring), derivation of a score 
by adding heterogeneous item scores, and a problem similar 


to that with the user involvement measure, and the use of 
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parametric statistics. Since the majority of the 
discipline’s studies appear to use questionnaires involving 
Likert-type designs, information systems research is 
vulnerable to these problems. 

Researchers failed to state, in 25 of 36 
questionnaires, whether they were of original design or 
adapted from previous studies. Not knowing the source of an 
instrument’s design, impedes evaluation of its validity. 
The failure to cite original design sources wrongly 
precluded readers from referencing discussions of validity 
issues which may have appeared in previous studies. 

4. Use of Computer Records 

Analyses of existing computer records accounted for 
13 of the 65 instruments. Few researchers are exploiting 
computer records, which might already reside on the 
investigated information systems, as the bases of their 
data. Rather than using objective measures provided by 
archival records, they are primarily turning to a secondary 
subjective measures, as provided by human respondents. 

5. Ambiguous Designs of Interviews 

Sufficient details were provided to make 
determinations regarding schedule and structure for only 
three of the nine interviews conducted. None of the 
information systems researchers provided sufficient 
discussion to develop an understanding of the manner in 


which the interviews were conducted and the questions 
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presented. Researchers did little more than identify that 
interviews were conducted, thus making interpretations of 
conduct and content virtually impossible. 

6. Obtrusive Role of Observers 

Only four of the 37 studies involved collection of 
data by observers. The researchers used participant 
observers in three studies and complete observers in one 
study. Participant observation is an obtrusive form of data 
collection and complete observation is an unobtrusive form 
of data collection. The more frequent reliance upon 
participant observation demonstrates a possible tendency for 
researcher to relieve more on obtrusive rather than 
unobtrusive forms of observation. 

Despite having used obtrusive forms of observation, 
none of the researchers assessed impacts of observers on 
studied information systems. Researchers failed to 
ëng that the introduction of observers impacted 
recorded measures. Although unobtrusive observation of 
information systems is clearly desirable over obtrusive 
observation, it is difficult for the observer to remain 
concealed and can be very time consuming when conducted 
outside the laboratory. This may explain why the majority 
of researchers who used observations selected the obtrusive 


form of participant observation. 
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B. FAILURE TO RECOGNIZE MEASURES OF ATTITUDE OR BELIEF 

Marshall and Rossman (1989) describes questionnaires as 
instruments that measure sets of attitudes or beliefs. 
Interviews also solicit the respondent to express attitudes 
or beliefs regarding phenomena of interest. Researchers who 
use questionnaires or interviews are, by the very nature of 
these instruments, measuring the attitudes or beliefs of 
respondents. 

The researchers used a total of 45 questionnaires and 
interviews. Only 27 of the 45 questionnaires and interviews 
were acknowledged to have measured attitudes or beliefs. 
Many researchers did not acknowledge or were unaware that 
they measured attitudes or briefs with questionnaires and 
interviews. 

Many researchers purported that they collected other 
types of measures with questionnaires and interviews. The 
45 questionnaires and interviews used by the researchers 
were purported to provide 37 measures of system 
characteristics, behavior, and situational characteristics. 
Many researchers erroneously believed questionnaires and 
interviews measure system characteristic, behavior, and 
Situational characteristics; when in fact, these instruments 


exclusively measure attitudes or beliefs. 
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C. ADMINISTRATION OF DATA COLLECTION INSTRUMENTS 
1. Samples of Convenience 

All 37 studies appear to involve samples selected on 
bases of convenience. The ability to generalize findings 
becomes questionable when sample are selected on the bases 
of convenience. The characteristics of such samples may not 
accurately reflect the population they are supposed to 
represent. Cochran (1953) contends, 

"The people from whom data are obtained, the sample, are 
of interest only insofar as the data tell us something 
about some larger group of people whom statisticians 
call the population or universe. Further, results 
obtained from a sample can be extended to a larger 
population with logical soundness only if the sample is, 
in a certain technical sense, a probability sample drawn 
from that population.” 

Samples selected on bases of convenience are not 
necessarily a probability sample drawn from the populations 
under study. Without any indications that selected samples 
represent the studied populations, researchers are 
generalizing the findings on their studies. 

2. Practitioners and Users as Respondents 

A total of 49 questionnaires, interviews, and 
observations were used as data collection instruments. 
Humans either responded to or were the subjects observed for 
all of these instruments. Practitioners and users of 
information systems were the focus of 43 of these 49 
instruments. The remaining 6 instruments involved students. 
The high proportion of studies involving practitioners and 


users of information systems, rather than college students, 
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enhanced the ability to generalize the findings of the 
studies onto operational situations. 
3. Insufficient Information Concerning Sample Size 
The researchers failed to provide information 
clearly identifying the number of samples, duration of 
Sampling, or frequency of collections for 27 of the 65 
instruments. The absence of such information prevents 
readers from fully assessing how well samples represented 
the studied populations. 
4. Insufficient Information Concerning Response Rates 
The questionnaires and interviews used in the 37 
studies all relied upon respondent participation. The 
representativeness of samples collected with these 
instruments is affected by the response rates achieved. The 
researchers failed to provide information regarding the 
response rates achieved for 20 of the 40 questionnaires and 
interviews. The absence of such information prevents 
evaluation of how well the samples represented the studied 
populations. 
5. Distribution of Questionnaires by Mail 
The researchers failed to provided information 
identifying how 18 of the 36 questionnaires used were 
distributed. Mail was the most common method of 
distribution, used for 15 of the 36 questionnaires. The 
researchers that mailed questionnaires were vulnerable to 


persons other than the identified respondents having 
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completed the questionnaires. Without face-to-face 
interaction, it is difficult to know who completes a 
questionnaire. Because the researchers do not know for 
certain the identities of the respondents, the 
representative of the samples becomes suspect. 

6. Few Researchers Discuss Non-Respondents 

The researchers failed to discuss differences between 
respondents and non-respondents for 30 of the 40 instruments 
involving questionnaires or interviews. The failure to 
identify differences precluded readers from assessing 
whether the researchers had addressed issues regarding 
respondent biases. The absence of discussions of 
differences between respondents and non-respondents raised 
doubts concerning the representativeness of almost all 
samples. 
D. HANDLING OF VALIDITY ISSUES 

1. Few Researchers Address Internal Validity Issues 

The researchers acknowledged deficiencies regarding 
the internal validity of six of the 65 data instruments. 
The acknowledgements of such deficiencies indicates that the 
researchers responsible for the six involved studies were 
aware and sensitive to internal validity issues. 
The researchers asserted another eight of the 65 

instruments demonstrated internal validity. The assertions 


of internal validity indicates that the researchers 
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responsible for the eight involved studies were also aware 
and sensitive to internal validity issues. 

Sufficient discussions of internal validity were not 
presented for the overwhelming majority of instruments. The 
researchers failed to present discussions of internal 
validity for 46 of the 65 instruments. Their discussions 
for the remaining five instruments cited secondary sources 
which provided little evidence supporting internal validity. 

In all, 51 of the 65 instruments used in the 37 
studies were not accompanied by adequate discussions of 
internal validity. Instruments are not considered to have 
demonstrated validity unless sufficient evidence is 
presented to support such claims. None of these 51 
instruments can be considered to have demonstrated internal 
validity. All of the researchers who used these instruments 
based their findings upon measures that have not 
demonstrated internal validity. This raises a question as 
to whether these researchers, most of those involved in the 
37 studies, ignored internal validity issues or were simply 
not aware of them. 

2. Few Researchers Address External Validity Issues 

The researchers acknowledged deficiencies regarding 
the external validity of six of the 65 instruments. The 
acknowledgements of such deficiencies indicates that the 
researchers responsible for the six involved studies were 


aware and sensitive to external validity issues. 
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The researchers responsible for four studies 
presented or cited discussions which asserted that their 
instruments had demonstrated external validity. Clearly, 
they were aware and sensitive to external validity issues. 

Without having provided discussions of external 
validity issues, the researchers generalized the findings of 
studies which were based upon data from six of the 65 
instruments. The absence of explicit discussions of 
external validity issues indicates that these researchers, 
responsible for seven of the studies, either ignored or were 
unaware of external validity issues. 

Sufficient discussions of external validity were not 
presented for the overwhelming majority of data collection 
instruments. The researchers failed to present discussions 
of external validity for another 43 of the 65 instruments. 
Their discussions regarding the remaining six instruments 
cited secondary sources which provided little evidence 
supporting external validity. 

In all, 55 of the 65 instruments used in the 37 
studies were not accompanied with adequate discussions of 
external validity. None of these 55 instruments can be 
considered to have demonstrated external validity. All of 
the researchers who used these instruments based their 
findings upon measures that have not demonstrated external 
validity. This raises a question as to whether these 


researchers, most of those involved in the 37 analyzed 
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studies, ignored external validity issues or were simply not 
aware of them. 

3. Few Researchers Identify Specific Types of 

Validity 

Researchers claimed or implied that 15 instruments 
demonstrated validity. Specific types of validity were 
identified for five instruments that were claimed to have 
demonstrated validity. The four studies, which used these 
instruments, identified combinations of the following 
specific types of validity: internal, external, content, 
construct, discriminant-convergent, and criterion. These 
were the only researchers who comprehensively confronted 
validity issues, having both claimed and identified specific 
types of validity. These researchers were aware of the 
different types of validity. 

The remaining ten data collection instruments were 
implied to have demonstrated validity. Their studies 
included no discussions which directly identified specific 
types of validity, rather they implied individual 
instruments had demonstrated internal and or external 
validity. Readers can only guess whether these researchers 


were aware of the different types of validity. 
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4. Few Researchers Identify the Sources of Claimed or 

Implied Validity 

Researchers claimed or implied that 15 instruments 
demonstrated validity. Researchers responsible for the four 
studies, which claimed that five instruments demonstrated 
validity, identified analyses of instrument measures as 
their sources of validity. These researchers openly stated 
that the measures from their individual instruments served 
as the sources for claims of validity. Readers can 
investigate these sources to substantiate the claims of 
validity, enhancing the credibility of the studies. 

The remaining ten data collection instruments were 
implied to have demonstrated validity. None of the ten 
studies involving these instruments identified sources of 
validity. The absence of cited sources precludes readers 
from substantiating the implied validity of these 
instruments, detracting from the credibility of the studies. 

5. Inadequacy of Collectively Addressing Validity 

Issues 

Eighteen studies used multiple data collection 
instruments. In 14 of these studies, researchers presented 
discussions that addressed validity issues in a collective 
manner. Such discussions provided no information regarding 
the validity of individual instruments, rather they solely 


addressed the collective validity of the entire studies. 
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Although discussions regarding collective validity were 
interesting to read, they were essentially meaningless in 
the absence of discussions that substantiated the validity 
individual instruments. Without having presented evidence 
that the individual instruments have demonstrated validity, 
researcher can not reasonably expect to claim or imply 
collective validity for entire studies. 

Many researchers appear to have mistakenly believed 
that discussions of only collective validity adequately 
addressed validity issues. Researchers often presented 
discussions of only collective validity in their studies. 
Only two of the 14 studies included discussions of both 
individual and collective validity. The absence of 
discussions substantiating the validity of individual 
instruments, necessitates that claims of collective validity 
be discounted for the 12 studies that included discussions 
of only collective validity. 

E. RECOMMENDATIONS 

A minority of researchers addressed validity issues. 
Discussions regarding the validity of data collection 
instruments were generally superficial. Rarely did 
researchers demonstrate awareness or sensitivity to validity 
issues. Most researchers appear to have expected 
unquestioned acceptance of their findings, since readers 
were provided with relatively few discussions of validity. 


Such expectations are inappropriate in scientific research. 
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The failure to discuss validity issues detracts from the 
credibility of research involving the information systems 
discipline. 

An emphasis on validity issues should be encouraged 
throughout the information systems discipline. Students, 
practitioners, and researchers must be made acutely aware 
and sensitive to the concept of validity. Implementation of 
the following suggestions would heighten awareness and 
sensitivity: 


1. Students should receive exposure to the concept of 
validity as part of their graduate school instruction. 


2. Practitioners should implement the recommendations 
resulting from only studies that discussed and 
demonstrated validity. 

3. Professional journals pertaining to the information 
systems discipline should require that all articles 
submitted by researchers include discussions of 
validity issues. 

Information systems researchers have adopted the data 
collection instruments commonly found in social science 
research, yet have largely failed to face-up to the validity 
issues that inherently arise with their use. Social 
scientists have long understood and addressed validity 
issues. Information systems researchers should confront 
validity issues in the same manner as social scientists. 
They must become aware of the threats to validity and 


develop experimental designs which enhance both internal and 


external validity. They must clearly and comprehensively 
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discuss validity issues, adding credibility to information 


systems research. 
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APPENDIX B 


DETAILED ANALYSIS OF STUDIES 


Colton, K. W., "Computers and Police: Patterns of Success 
and Failure," Sloan Management Review, Vol. 14, No. 2, 
(Winter 1972-73), pp. 75-98. 


Description of System Studied: Information systems used by 
police departments. Applications included police patrol and 
daily operations; investigative analysis; administration and 
management; and program, planning, and evaluation. 


Data Collection Technique (s) Used: All respondents 


completed questionnaire. Selected respondents were later 
interviewed. 


Characteristics of Measuring Instruments Used: Specific 
characteristics of the measuring instruments were not 


explicitly discussed. The survey was devised by the 
International City Management Association, appearing not to 
have complied with an established format. No formal 
questionnaire was administered, but an effort was made to 
ask a common set of questions--an unstructured and scheduled 
interview. 


Objective(s) of Measurement Process: Questionnaires were 


aimed at documenting the growth, extent, and impact of 
computers used by police departments. Interviews were aimed 
at evaluating questions on impact, effectiveness, and 
success. Data collected were not specifically identified as 
measures of attitude, opinion, behavior, or etc.. 


Participants in Measurement Process: Police chiefs and data 
processing managers of police departments in cities with 
populations of over 25,000. 


Distribution of Measuring Instruments: Questionnaires were 
mailed to all police departments in cities with populations 


of over 50,000 and to 25% of cities with populations between 
25,000 and 50,000. A total of 498 questionnaires were 
mailed and 376 responses were received (75.5%). Interviews 
were personally conducted at 14 police departments. Visits 
of 3 to 4 days were conducted at 9 police departments and 
brief contacts were made in 5 other departments. A total of 
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143 interviews were conducted with law enforcement 
personnel. 


Corrective Action taken to Account for Non-participants: 
Second mailings to cities that failed to respond were made. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Method of data analysis not 
stated. 


Issue of Validity Addressed: Not discussed. 
Type of Validity Claimed: Not discussed. 


Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative, Metric, Interval, 
Frequency, and Ordinal. 
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Powers, R. F. and G. W. Dickson, "MIS Project Management: 
Myths, Opinions and Reality," California Management Review, 
Vol. 15, No..3, (1973)) peels loo 


Description of System Studied: Twenty management 
information systems used in the manufacturing, wholesale and 


retail trade, transportation, finance, utility, and 
commodity merchandising and processing businesses. 


Data Collection Technique(s) Used: Questionnaire for 


initially identifying factors affecting MIS project success. 
Subsequent interview and questionnaire to research 
correlates of success with MIS projects. 


Characteristics of Measuring Instruments Used: The 
questionnaire used to initially identify factors affecting 


MIS project success required the respondent to rank the 
importance of 35 factors, as selected by 25 MIS experts, on 
a five point scale. Only the average time spent during each 
interview was provided, 9 hours per firm. No information 
was provided pertaining to the second questionnaire, beyond 
the fact that it was pretested. 


Objective(s) of Measurement Process: Determine what 


organizational and procedural factors are correlates of 
success with MIS projects. The questionnaire used to 
initially identify factors affecting MIS project success was 
acknowledged as measuring beliefs. Data collected in 
interviews and with second questionnaire were not 
specifically identified as measuring beliefs, attitudes, 
behavior, or etc.. 


Participants in Measurement Process: Initial questionnaire 
was given to MIS experts attending the Founding Conference 


of the Society for Management Information Systems at the 
University of Minnesota in 1969. The interviews and second 
questionnaires were used to study 10 firms, with each 
providing 2 MIS projects, in the Minneapolis and St. Paul 
area. The respondents within the firms were directors of 
MIS, project leaders, computer operating personnel, and 
management users. 


Distribution of Measuring Instruments: The initial 


questionnaire was given to the respondents. The interview 
and second questionnaire was conducted on-site, at the 
firms. Response rates were not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 
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Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Data were analyzed using non- 
parametric statistical techniques, specifically Kendall’s 
tau statistic. 


Issue of Validity Addressed: "Kendall’s tau statistic was 
chosen for the data analysis because of its statistical 
properties which facilitated testing the various hypotheses 
for significance." The ability to generalize findings was 
briefly discussed (External validity). 


Type of Validity Claimed: External validity, the authors 
stated that in their opinions their findings could be 
generalized to MIS projects. 

Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative, Metric, Interval, and 
Frequency. 


93 


Lucas, H. C., Jr., "System Quality, User Reactions and The 
Use of Information Systems," Management Informatics, Vol. 3, 
No. 4, (1974), pp. 207-212. 


Description of System Studied: Twenty-six separately 
identifiable batch computer systems at a major university. 


These systems are concerned with accounting, student record 
keeping, purchasing, fund raising, alumni record keeping, 
etc.. 


Data Collection Technique(s) Used: Two questionnaires. 


Characteristics of Measuring Instruments Used: The first 
questionnaire was similar in structure to those used in past 


studies, the majority of responses have a range from one to 
seven. The second questionnaire was a structured 
questionnaire. 


Objective(s) of Measurement Process: The first 


questionnaire measured user attitudes toward information 
services department activities and perceptions of the 
quality of information systems service were measured. The 
second questionnaire dealt with the quality of systems as 
seen by users. 


Participants in Measurement Process: The first 
questionnaire was completed by 117 administrative users of 


the 26 systems at the major university. The second 
questionnaire was completed by staff members in operations 
and systems design positions. The second questionnaire was 
completed by between 1 and 3 staff members in the 
information services department, on each of the 26 systems. 


Distribution of Measuring Instruments: Methods of 
distribution were not discussed. An adjusted response rate 


of 70% for those distributed was claimed for the first 
questionnaire. No response rate was discussed for the 
second survey. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Step-wise multiple regression 
used to analyze the data. 
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Issue of Validity Addressed: The author discussed extension 
of study: "The research techniques and instruments in this 
study can be used by any organization to evaluate its 
information systems. This study has demonstrated an 
association between systems quality and user reactions to 
information systems.... Though causality has not been 
demonstrated, a priori arguments and data support the goals 
of developing favorable user attitudes and high quality 
systems to encourage use of information systems." 


Type of Validity Claimed: External validity through 
argument of extension. 


Source of Validity Claimed: A priori arguments and data. 
Type of Data Recorded: Ordinal. 
Secondary Source(s) Cited: Lucas (1973) is where the 


original questionnaire was presented, however, its validity 
was not clearly established. 
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Swanson, E. B., "Management Information Systems: 
Appreciation and Involvement," Management Science, Vol. 21, 
No. 2, (October 1974), pp. 178-188. 


Description of System Studied: An activity reporting system 
(MIS) used by a 200 employee department of a large 


international manufacturer of complex electronic equipment. 
The system gathers data on the planned and actual work 
activity of the department’s members, and makes it available 
to management on a "need to know" basis. 


Data Collection Technique(s) Used: Analysis of existing 


documents and records and questionnaire. 


Characteristics of Measuring Instruments Used: Computer 


logs were the archival data analyzed. The questionnaire 
consisted of a set of 16 items, based upon an arbitrary 0.0 
to 1.0 scale range, subdivided into five intervals, assumed 
to be equal. The respondent checked an interval. The 
questionnaire consisted of a set of 10 items, based upon a 
frequency range. 


Objective(s) of Measurement Process: Archival data used to 


measure system usage. The research questionnaire was 
constructed to obtain indicators of MIS appreciation and a 
priori involvement. Data collected in questionnaire was not 
specifically identified as measuring beliefs, attitudes, 
behavior, or etc.. 


Participants in Measurement Process: Users of the system. 
Distribution of Measuring Instruments: Method of 


distribution appears to have been mail. Of the 46 
questionnaires distributed, 37 were returned. 


Corrective Action taken to Account for Non-participants: 


Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Simple classification and non- 
parametric analysis of data. 
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Issue of Validity Addressed: "It should be noted, however, 
that the direction of the original causal argument has not 


been tested... no controlled experimentation was undertaken 
in this study.... In short, there is no beginning for a 
causal argument." Later, "The single case presented here 


does not permit effective generalization." 
Type of Validity Claimed: None. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal and frequency (using an 
ordinal scale). 
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Gallagher, C. A., “Perceptions of the Value of a Management 
Information System," Academy of Management Journal, Vol. 17, 
No. 1, (March 1974), pp. 46-55. 


Description of System Studied: A management information 
system used for cost accounting. It was developed in 
1967/68 and used by a company employing 1,800 people, of 
whom 375 were management personnel. 


Data Collection Technique(s) Used: Questionnaire. 


Characteristics of Measuring Instruments Used: 
Questionnaire consists of two parts. The first part 


solicited an opinion expressed in dollar values. The second 
part solicited an opinion reported by scoring fifteen 
bipolar adjective pairs on a seven-point scale. 


Objective(s) of Measurement Process: The study was used to 


determine the value of an existing set of MIS reports. The 
study measures user perceptions. 


Participants in Measurement Process: Managers using reports 
generated by the system. 


Distribution of Measuring Instruments: The questionnaire 
was mailed to 103 managers, of which 75 usable replies were 


received. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Semantic differential results 
and correlation between value measures were used to analyze 
the data. 


Issue of Validity Addressed: "The principal challenge lies 
in designing questions which will minimize the effects of 
bias and inaccuracy in user perceptions.... This study 
included one test of a new methodology. The results of a 
single test cannot establish the validity of a methodology, 
although they can discredit all or part of the 
methodology.... A question may be raised regarding the use 
of hypothetical questions. Do the answers to a hypothetical 
question bear any relationship to those that would be 
obtained in an actual situation? Surely the answers would 
not be identical." 
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Type of Validity Claimed: None 


Source of Validity Claimed: Study identified lack of 
validity. 


Type of Data Recorded: Interval and ordinal. 
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Lucas, H. C., Jr., "Performance and The Use of an 
Information System," Management Science, Vol. 21, No. 8, 
(April 1975a), pp. 908-919. 


Description of System Studied: An information system used 
to provide sales activity data for a major manufacturer of 
ready-to-wear clothing. 


Data Collection Technique(s) Used: Analyses of existing 
récords and documents and questionnaire. 


Characteristics of Measuring Instruments Used: Computer 
records and personnel files are the existing records used in 


the study. The questionnaires employed scales ranging from 
l to 5 or Ptol: 


Objective(s) of Measurement Process: The goals of the study 


was to determine the variables influencing sales force 
performance and the use of a sales information system and to 
determine if a relationship exists between the use of the 
system and performance. Data collected by analyses of 
computer files measures performance and situation. Data 
collected by analyses of personnel records measures personal 
variables. Data collected by the questionnaires were 
measures of usage, attitudes, and perceptions. 


Participants_in Measurement Process: Members of the sales 


force (salesmen and account executives). 


Distribution of Measuring Instruments: Distribution method 
unknown. A questionnaire was administered to the 439 


members of the sales force; approximately a 90% return rate 
was achieved (316 salesmen and 82 account executives). 


Corrective Action taken to Account for Non-participants: 
One follow-up. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Step-wise multiple regression 
analysis was used to analyze the data. 


Issue of Validity Addressed: "It is difficult to draw 
conclusions about causality from cross-sectional studies. 
Both laboratory studies with their superior controls and 
field studies are needed to confirm and expand the findings 
reported here." 


Type of Validity Claimed: None. 
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Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Ordinal and other unknown. 
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Lucas, H. C., Jr., "The Use of Accounting Information 
System, Action and Organizational Performance," Accounting 
Review, (October 1975b), pp. 735-746. 


Description of System Studied: Information system used by a 
major California bank with more than 200 branches in the 


state. The information system was used to generate 
accounting information. 


Data Collection Technique (s) Used: Analyses of existing 


records and documents and questionnaires. 


Characteristics of Measuring Instruments Used: Computer 
files were used for analyses of existing records and 


documents. Questionnaires employed scales. 


Objective(s) of Measurement Process: The goals of the study 


were to determine what variables are associated with branch 
performance and specify what variables are associated with 
action and the use of the accounting information system. 
Data obtained from computer files were used to measure 
performance and situation. Questionnaires were used to 
measure attitude. 


Participants in Measurement Process: A questionnaire was 


administered to managers and assistant managers in 165 
branches in three geographic divisions of the bank. Another 
questionnaire was administered to division managers. 


Distribution of Measuring Instruments: The questionnaire 
completed by branch managers and assistant managers was 


mailed. A 96% return rate was achieved for managers and a 
95% return rate was achieved for assistant managers. 
Distribution and response rate of questionnaire administered 
to the division managers is unknown. 


Corrective Action taken to Account for Non-participants: 


Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Step-wise multiple regression 
was used to analyze the data in the study. 
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Issue of Validity Addressed: Factor analysis and 
correlation coefficients were used to determine if items 


could be combined in a single scale during the development 
of the questionnaire. "In a field study it is difficult to 
obtain good measures for operationalized variables and to 
control for confounding variables. The lack of extremely 
strong results can be partially attributed to errors in 
measuring the variables which were included in the 
research." 


Type of Validity Claimed: None. 
Source of Validity Claimed: None. 
Type of Data Recorded: Ordinal and unknown other (s). 


Secondary Source(s) Cited: Lucas (1973) is where the 


original questionnaire was presented, however, its validity 
was not clearly established. 
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Hall P. G. and T. J. Lincoln, "A Cost Effectiveness Study of 
Production and Engineering Systems within a Tightly 
Integrated Manufacturing Environment," Management 
Datamatics, Vol. 5, No. 6, (1976), pp. 262-274. 


Description of System Studied: Production and engineering 
computer systems in the Commercial Aircraft Division of 


British Aircraft Corporation. Applications involving 
product definition, work in progress, and supplies 
management were performed on these systems. 


Data Collection Technique(s) Used: Participant observation 


and interviews. 


Characteristics of Measuring Instruments Used: The study 
began with a two days of observing the system in question 


and interviewing the senior systems analyst. One primary 
user of each system was interviewed for 1 to 2 hours. User 
management was interviewed after receiving documentation 
resulting from the primary user interview. It was made 
clear that the user management interview would form the 
basis for further discussions with senior management. A 
final presentation and interview of senior management was 
conducted prior to presentation of results to the company’s 
board. 


Objective(s) of Measurement Process: Interview with the 


senior system analyst provided computer cost and background 
information regarding system. Interview with primary users 
identified user costs and the impact of the system on the 
user’s department. Interview of user management provided 
costs and benefits information. Presentation and interview 
of senior management was aimed at informing management of 
results and ironing out any difficulties or uncertainties 
which still existed. Data collected were not specifically 
identified as measurements of attitudes, opinion, behavior, 
or etc.. 


Participants in Measurement Process: Senior systems 
analysts, one primary user of each system (as identified by 
its senior systems analyst), user management, and senior 
management. 


Distribution of Measuring Instruments: All interviews 
appear to have been conducted in person. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 
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Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Method of data analysis not 
stated. 


Issue of Validity Addressed: Essentially stated that other 
companies should conduct similar studies and generalizes its 
findings. 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Interval, metrics, and unknown 
others. 
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Schewe, C. D., "The Management Information System User: An 
Exploratory Behavioral Analysis," Academy of Management 
Journal, Vol. 19, No. 4, (December 1976), pp. 577-590. 


Description of System Studied: Eight batch and two 
interactive systems of ten food processing firms in three 


midwestern states. 


Data Collection Technique(s) Used: Questionnaire and 


possible analysis of existing records and documents. 


Characteristics of Measuring Instruments Used: 


Questionnaire used a five-point, bipolar scale. No details 
regarding how system usage (monthly requests) were measured. 


Objective(s) of Measurement Process: "This study focused on 


attitudes toward use of the information system." "System 
usage in this study was measured by the number of monthly 
requests that a manager/system user made for additional 
information." "A behavioral study utilizing an attitudinal 
model explored the relationships between MIS users’ 
perceptions of their computer system, perceived variables 
exogenous to the system, attitudes, and system usage." 


Participants in Measurement Process: Middle managers (41 
batch system users and 38 interactive system users). 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Data were analyzed by the use 
of step-wise regression. 


Issue of Validity Addressed: "This study was exploratory, 
and the attitude variables studied are not completely 


comprehensive of the system user’s attitude system toward 
usage of the MIS." 


Type of Validity Claimed: None. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal. 
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Edstróm, A., "User Influence and the Success of MIS 
Projects: A Contingency Approach," Human Relations, Vol. 
Sereno. 7, (1977), pp: 589-607. 


Description of System Studied: Systems designed to support 
management decision-making, selected from sixteen companies. 


All companies were located in the Paris region of France. 
The companies were involved in manufacturing, 
transportation, and service industries. 


Data Collection Technique (s) Used: Interviews. 


Characteristics of Measuring Instruments Used: Structured 
interviews with four key players in each organization. The 


interview questions have been of both the open ended and 
fixed alternative answer kind. A seven point rating scale 
was used. 


Objective(s) of Measurement Process: "It should be pointed 


out, however, that the measures of outcomes are perceptual 
measures rather than assessments based on objective 
criteria." The data were measures of perceptions. 


Participants in Measurement Process: System development 
manager, functional manager, project manager, and a user. 


Distribution of Measuring Instruments: Personal interviews. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Spearman-rank correlation 
coefficients were used to analyze the data. 


Issue of Validity Addressed: "There are several weaknesses 
in the proposed design that need mentioning. First, we are 
relying on people’s memories for data on the system- 


development process.... A second weakness in our design is 
the fact that we have to rely on the perceptions of people 
instead of making direct observations.... Third, our 


results are limited to the particular roles in the 
development process that we studied. Application to systems 
development in general must be made with caution, since we 
only have one informant per group." 
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Type of Validity Claimed: None. 
Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Ordinal. 
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King, W. R. and J. I. Rodriguez, "Evaluating Management 
Information Systems," Management Information System 
Quarterly, Vol. 2, No. 3, (September 1978), pp. 43-51. 


Description of System Studied: A system designed to aid 
managers in understanding and resolving competitor-related 


strategic issues. It was a system that combined an 
intelligent management information system and a management 
decision support system. 


Data Collection Technique(s) Used: Experiment using a 


questionnaire. 


Characteristics of Measuring Instruments Used: Experiment 
conducted in a simulated business environment. True 


experiment, pre/post test and post test only, control group 
design. Questionnaire was developed by Schultz and Slevin, 
employing ordinal scale. 


Objective(s) of Measurement Process: The questionnaire 


assessed attitudes and value perceptions. 


Participants in Measurement Process: Forty-five experienced 
manager-users, who were enrolled in part-time MBA program. 


Distribution of Measuring Instruments: Questionnaire given 
to students participating in MBA program. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Methods used to analysis data 
were Schultz-Slevin "dependent variables", Kendall Rank 
Correlation test, and possible regression. 


Issue of Validity Addressed: "Clearly, behavior in the 
Simulated business environment in which the evaluation was 
performed may not be closely related to behavior in the real 


world which may raise questions as to the external validity 
of the experiment." 


Type of Validity Claimed: None. 


Source of Validity Claimed: Not discussed. 
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Type of Data Recorded: Ordinal. 


Secondary Source(s) Cited: Schultz and Slevin (1975) is 
where the development of the original instrument was 
presented, however, its validity was not clearly 
established. 
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Elchesen, D. R., "Cost-Effectiveness Comparison of Manual 
and On-line Retrospective Bibliographic Searching," Journal 
of American Society for Information Science, Vol. 29, No. 2, 
(March 1978), pp. 56-66. 


Description of System Studied: Manual and on-line 
bibliographic information. Forty search queries were 


processed against seven abstracting-indexing publications 
and the corresponding SDC/ORBIT data base. 


Data Collection Technique(s) Used: Experiment relying on 


observation of participants. 


Characteristics of Measuring Instruments Used: Participants 
measured variables of interest and recorded results on logs. 


Participants assigned values to the effectiveness of the 
search results. 


Objective(s) of Measurement Process: Measures of 


performance were implied. Data collected were not 
specifically identified as measures of attitude, opinion, 
behavior, or etc.. 


Participants in Measurement Process: Eight of thirteen 
members of the Lawrence Livermore Laboratory Research 


Information Group. The group was charged with providing 
bibliographic information service to the Laboratory’s 
scientific and engineering staff. All were experienced with 
manual and on-line bibliographic searching. 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Descriptive statistics were 
used to analyze the data. 


Issue of Validity Addressed: Researchers generalize results 
of study. 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative and metric. 
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Robey, D. and R. L. Zeller, "Factors Affecting the Success 
and Failure of an Information System for Product Quality," 
Interfaces, Vol. 8, No. 3, (February 1978), pp. 70-75. 


Description of System Studied: Quality information system 
used by the product quality section of two neighboring 
plants of a division of a very large U.S. corporation. The 
divisions manufactured a wide variety of highly technical 
equipment of the health care industry. 


Data Collection Technique(s) Used: Interview, supplemental 


questionnaire, analysis of existing records and documents, 
and interviews. 


Characteristics of Measuring Instruments Used: No 
information regarding the conduct of the interview. Company 


documents were used during the analysis of existing 
documents and records. Questionnaire was developed by 
Schultz and Slevin. 


Objective(s) of Measurement Process: Interviews gathered 


general information about the implementation process in each 


department. The interview and company documents were used 
to identify some differences in structure between the 
departments. Questionnaires measured attitudes. "Responses 


show the user’s perceptions of how favorable the system is 
for each area of concern." 


Participants in Measurement Process: Eleven users. 
Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Mann-Whitney U test was used 
to analyze the data. 


Issue of Validity Addressed: "While we cannot say that 
attitudes were the cause of behavior in this case, certain 
attitudes, and not others, are related to behavior." "The 
essential advantage of our research design is its ability to 
compare attitudes, organization structure and implementation 
process while holding the MIS features, production 
technology and corporate policy constant. By controlling 
the effects of these other factors, the effects of the 
behavioral and organizational factors can be more readily 


dE 


identified." 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal. 


Secondary Source(s) Cited: Schultz and Slevin (1975) is 
where the development of the original instrument was 
presented, however, its validity was not clearly 
established. 
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Lucas, H. C., Jr., “Unsuccessful Implementation: The Case 
of a Computer-Based Order Entry System," Decision Sciences, 
Vol. 9, No. 1, (January 1978), pp. 68-79. 


Description of System Studied: Optical character 
recognition system and keypunch system for order entry 


procedures conducted in two divisions of a large West Coast 
apparel manufacturing company. 


Data Collection Technique (s) Used: Experiment and 


questionnaire 


Characteristics of Measuring Instruments Used: Experiment 
employed a control group and examined the implementation of 


a system in a longitudinal field study. The questionnaire 
used scaled variables form 1 to 7. Modified version of the 
JDI work scale was used to measure satisfaction. 


Objective (s) of Measurement Process: Questionnaire measures 


success of implementation, costs, and benefits. 
Participants in Measurement Process: Sales representatives. 


Distribution of Measuring Instruments: Distribution to the 
experimental group (approximately 120 sales representative) 
is unknown. Questionnaires were mailed to the control group 
of 70 randomly selected sales representatives. 

Approximately three months later a follow-up questionnaire 
was mailed. Response to the second questionnaire was 56 for 
the experimental group and 58 for the control group. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Dynamic and cross-lagged 
correlational analysis was employed. 


Issue of Validity Addressed: "The quasi-experimental 
research design and these statistical techniques provided 
increased confidence about causal inferences concerning the 
relationship between the benefits and costs of the new 
system and successful implementation." 


Type of Validity Claimed: Internal (implied). 


Source of Validity Claimed: Not discussed. 
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Type of Data Recorded: Ordinal. 
Secondary Source(s) Cited: Smith, Kendall, and Hulin (1969) 


present original instrument and established its validity. 


PIS 


Neumann, S. and E. Segev, "A Case Study of User Evaluation 
of Information Characteristics for Systems Improvement, " 
Information & Management, Vol. 2, (1979), pp. 271-278. 


Description of System Studied: A computerized information 
system which served 148 branches of one of the three leading 
banks in Israel. 


Data Collection Technique(s) Used: Interview and 


questionnaire were administered jointly. 


Characteristics of Measuring Instruments Used: The 
interview was unstructured and scheduled. A 1 to 5 scale 


was used in the questionnaire. 


Objective(s) of Measurement Process: The data were measures 


of the respondents perceptions. 


Participants in Measurement Process: Branch managers. 


Distribution of Measuring Instruments: Interviews conducted 
in person. Questionnaires were provided by hand to 130 


branch managers. Eighty-one sets of interviews and 
questionnaires were properly completed and returned by mail. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Guttman’s correlation 
coefficients of weak monotonicity. 


Issue of Validity Addressed: "We believe the findings may 
have implications for other organizations.... The findings 
have no implications on the a priori users’ definitions of 
their needs, only on perceptions of existing systems.... 
Further research in other organizations,... is necessary to 
validate whether the findings of this study are unique or 
typical of other organizations." 


Type of Validity Claimed: None. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal. 
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Robey, D., "User Attitudes and Management Information System 
Use," Academy of Management Journal, Vol. 22, No. 3, 
(September 1979), pp. 527-538. 





Description of System Studied: Computer based system to 
record, update, and maintain information pertaining to 


customer sales accounts. The system was used by sales 
members of a large industrial products manufacturer. 


Data Collection Technique(s) Used: Analysis of existing 


documents and records and questionnaire. 


Characteristics of Measuring Instruments Used: Two kinds of 
account records were the existing documents and reports 


used. A Schultz and Slevin type of questionnaire was used. 
Both a five point and a ten point scale was used in the 
questionnaire. 


Objective(s) of Measurement Process: Account records were 


analyzed to measure system usage. The questionnaire 
measured users” attitudes. 


Participants in Measurement Process: Members of the sales 
force. 


Distribution of Measuring Instruments: Corporate mail was 
used for distribution and a 100% return rate was achieved. 


Corrective Action taken to Account for Non-participants: 
Not required. 


Identification of Differences between Participants and Non- 
participants: No non-participants. 


Method Used to Analyze Data: Cronbach's coefficient alpha, 
Spearman rank correlations, and Pearson product-moment 
correlations. 


Issue of Validity Addressed: "In returning to the more 
direct implications of this research, it is important to 
note its limitations. First, although the Schultz and 
Slevin instrument appears to be superior to single-item 
scales used elsewhere, it has not received extensive 
validation.... Second, a word of caution must be extended 
to those interpreting these results and the results of any 
cross-sectional study. Strong positive relationships have 
been demonstrated between specific user attitudes and actual 
use of an MIS. It does not necessarily follow, however, 
that attitudes cause behavior." 
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Type of Validity Claimed: None. 

Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Ordinal, frequency, and interval. 
Secondary Source(s) Cited: Schultz and Slevin (1975) is 
where the development of the original instrument was 


presented, however, its validity was not clearly 
established. 
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Mansour, A. H. and H. J. Watson, "The Determinants of 
Computer Based Information System Performance," Academy of 
Management Journal, Vol. 23, No. 3, (September 1980), pp. 
mea O33. 


Description of System Studied: Computer based information 
systems in engineering, finance, manufacturing, 
retail/wholesale, service, and government. 


Data Collection Technique(s) Used: Two questionnaires. 
Characteristics of Measuring Instruments Used: User 


questionnaire used a five point scale. Expert questionnaire 
was used for design of experiment. 


Objective(s) of Measurement Process: Questionnaire 


completed by users obtained data on behavioral, structural, 
environmental, and performance variables. Questionnaires 
completed by experts identified study variables and assigned 
weights to the variables. 


Participants in Measurement Process: Users and expert panel 
(three practitioners and three academicians in the CBIS 


field). 


Distribution of Measuring Instruments: Unknown quantity of 
user questionnaires were mailed. A total of 108 


questionnaires were returned. Distribution of expert 
questionnaire is unknown. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Step-wise regression. 


Issue of Validity Addressed: Author implies internal 
validity and external validity. 


Type of Validity Claimed: Internal and external (implied). 


Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal, qualitative, and interval. 
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Ball, L. D., “Increasing the Manager’s Productivity Through 
the Information System," Public Productivity Review, Vol. 4, 
No. 1, (March 1980), pp. 51-62. 


Description of System Studied: Reporting and retrieval 
software system that facilitated the specification and 


retrieval of data existing or derived from available, 
perhaps multiple, computer files. 


Data Collection Technique (s) Used: Analysis of existing 


documents and reports and questionnaire. 


Characteristics of Measuring Instruments Used: Report 
requests were the documents and reports analyzed. 


Questionnaires type was not discussed. 


Objective(s) of Measurement Process: Reports were analyzed 


to measure system usage. Apparently, the questionnaire 
solicited opinions of the managers concerning the system. 
Data collected were not specifically identified as measures 
of attitude, opinion, behavior, or etc... 


Participants in Measurement Process: Managers who were 


users of the system were interviewed. 


Distribution of Measuring Instruments: Not discussed beyond 
stating that the sample was very small. 


Corrective Action taken to Account for Non-participants: 


Not discussed. 


Identification of Differences between Participants and Non- 
participants: States possible reasons some managers were 


non-users of the system, thus making them non-participants 
in the study. 


Method Used to Analyze Data: Descriptive statistics. 
Issue of Validity Addressed: Not discussed. 
Type of Validity Claimed: Not discussed. 


Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative, Metric, Interval, and 
Frequency. 


120 


Edelman, F., "Managers, Computer Systems, and Productivity," 
Management Information System Quarterly, Vol. 5, No. 3, 
(September 1981), pp. 1-19. 


Description of System Studied: Industrial Relations 
Information System at RCA, used to provide MIS support 


relating to human resource management. The system included 
data base, interactive, office support, and management 
support technologies. 


Data Collection Technique (s) Used: Controlled experiment 


and apparently analysis of existing records and documents. 


Characteristics of Measuring Instruments Used: Experiment 
employed a target and control group. Apparently archival 


records pertaining to personnel costs were used during the 
experiment. 


Objective(s) of Measurement Process: Apparently the data 


collected measured costs. 


Participants in Measurement Process: Fourteen of RCA’s 
eighteen operating units. 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
None required. 


Identification of Differences between Participants and Non- 
participants: Clearly identified. 


Method Used to Analyze Data: Descriptive statistics. 


Issue of Validity Addressed: Only the validity of the 
experimental design used. 


Type of Validity Claimed: Not discussed. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Interval and frequency. 
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Altmeyer, A. S. and L. S. Bozeman, "Productivity via 
Computer Technology," Public Productivity Review, Vol. 5, 
No. 4, (December 1981), pp. 321-334. 


Description of System Studied: Welfare Management System, 
an information system designed to receive and process 


information on clients who have applied for or have been 
determined eligible for public assistance. Medicaid 
Management Information System, an information system 
designed to improve the availability of health care services 
as well as administrative controls over rising costs. 


Data Collection Technique(s) Used: Interviews and two 


questionnaires. Interviews and first questionnaire was used 
to design experiment. 


Characteristics of Measuring Instruments Used: A series of 


open-ended, unstructured field interviews. Semi-structured 
six-question telephone questionnaire, which asked open-ended 
questions regarding performance benefits. Survey 


questionnaire was self-administered, containing 102 
questions. The questionnaire required the responded to 
select "agree, do not know, or disagree" for statements. It 
was reviewed by a panel of experts, before administration. 


Objective(s) of Measurement Process: "Responses (of survey 


questionnaires) showed perceptional judgements of middle 
management...." Author presents findings gathered from 
survey data regarding managerial aspects; productivity; 
organizational effectiveness; and several user attitudes. 


Participants in Measurement Process: Interviews were 
conducted with personnel serving in several levels of county 


social services departments. Additional interview were held 
with a number of county commissioners as well as officials 
at the state office in Albany. The telephone questionnaire 
was administered to county staff developmental personnel. 
Survey questionnaire was administered to various levels of 
managers. 


Distribution of Measuring Instruments: Interviews were 
conducted in person, no response rate was provided. 


Telephone questionnaire was administered over the telephone, 
with 44 of 56 staff development coordinators responding. 
Survey questionnaires were mailed to 492 managers, with a 
response rate of 67%. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 
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Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Descriptive statistics. 


Issue of Validity Addressed: Results of study were 
generalized. 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative. 
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Ginzberg, M. J., "Early Diagnosis of MIS Implementation 
Failure: Promising Results and Unanswered Questions." 
Management Science, Vol. 27, No. 4, (April 1981), pp. 459- 
478. 


Description of System Studied: On-Line Portfolio Management 
system, an information system used to support portfolio 
managers in their job of managing asset portfolios 
consisting primarily of common stock, debt instruments, and 
cash. The system was operated in the trust department of a 
large U.S. bank. 


Data Collection Technique(s) Used: Interview, two 


questionnaires, and analysis of existing documents and 
records. 


Characteristics of Measuring Instruments Used: Interview 
used to develop pre-implementation questionnaire. Pretest 


of the pre-implementation questionnaire was conducted. Pre- 
implementation questionnaire consisted of open-ended 
questions. Post-implementation questionnaire used closed- 
ended questions and scales questions. Post-implementation 
analysis of existing documents and records was collected by 
systems software to measure system individual system usage. 


Objective(s) of Measurement Process: Author stated that 


three major types of data were collected: 1) users’ pre- 
implementation expectations, 2) expert group pre- 
implementation expectations and 3) users’ post- 
implementation assessment of success. Post-implementation 
data were collected to measure attitudes and behavior. 
Attitudes were measured by questionnaire and behavior by 
archival records. 


Participants in Measurement Process: Interviews to develop 
pre-implementation questionnaire were conducted with a 


sample of portfolio managers (15%), system developers (both 
internal and external), trust department management, and 
other trust department personnel. Pre-implementation 
questionnaire was administered to portfolio managers and 
others. Post-implementation questionnaire was administered 
to portfolio managers who had responded to the pre- 
implementation questionnaire. 
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Distribution of Measuring Instruments: Interviews appear to 
have been conducted in person. Pre-implementation interview 


was mailed to 44 portfolio managers and 21 others, with 86% 
and 82% response rates achieved respectively. Post- 
implementation interview was mailed to the portfolio 
managers and a 82% response rate was achieved. 


Corrective Action taken to Account for Non-participants: 
Follow-up message. 


Identification of Differences between Participants and Non- 
participants: Differences were discussed. "Analysis of the 


available data shows no significant, nor even sizable, 
difference between respondents and non-respondents on pre- 
implementation demographic or attitudinal measures." 


Method Used to Analyze Data: Pre-implementation expectation 
data were analyzed using McQuitty’s Elementary Linkage 
Analysis. Post-implementation data were analyzed using 
Kendall’s tau correlations. 

Issue of Validity Addressed: "While there is no a priori 
reason to believe that the specific characteristics of this 
situation limit the generalizability of the results, there 
is also no evidence to support such generalizations." 

Type of Validity Claimed: Limited external (implied). 


Source of Validity Claimed: None. 
Type of Data Recorded: Ordinal and frequency. 
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Cron, W. L. and M. G. Sobol, "The Relationship Between 
Computerization and Performance: A Strategy for Maximizing 
the Economic Benefits of Computerization," Information € 
Management, Vol. 6, No. 3, (June 1983), pp. 171-181. 


Description of System Studied: Computers used by 
independent wholesalers involved in the medical supply 


industry. 


Data Collection Technique(s) Used: Questionnaire. 


Characteristics of Measuring Instruments Used: Not 
specifically discussed. 


Objective(s) of Measurement Process: "The 1979 survey 


(questionnaire) was selected for analysis because it 
contained relevant computer usage information not in other 
editions." Data collected were not specifically identified 
as measuring of attitude, opinion, behavior, or etc.. 


Participants in Measurement Process: Wholesalers that 
responded to the annual financial and operating survey 


sponsored by the American Surgical Trade Association. 


Distribution of Measuring Instruments: Distribution is 
unknown. A total of 138 wholesalers responded, achieving a 


90% response rate. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Major non-participants identified and 
differences discussed. 


Method Used to Analyze Data: Descriptive statistics, Chi- 
square analysis, and analysis of variance. 


Issue of Validity Addressed: "Though it is always difficult 
to establish causality from correlation data conclusively, a 
number of significant relationships were found.... While 
recognizing that it is important to replicate this study 
using different kinds of firms and industries, results of 
this study indicate that computerization is positively 
related to the overall performance of medium and larger size 
firms." 


Type of Validity Claimed: External (implied). 


Source of Validity Claimed: Not discussed. 
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Type of Data Recorded: Qualitative, Interval, and possible 
others. 
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Bruwer, P .J .S., "A Descriptive Model of Success for 
Computer-Based Information Systems," Information € 
Management, Vol. 1, No. 2, (July 1984), pp. 63-67. 


Description of System Studied: 140 computerized systems, 
used by about 1200 clerical staff and 114 managers, ina 


very large organization. 


Data Collection Technique(s) Used: Questionnaire. 


Characteristics of Measuring Instruments Used: Most of the 
questions used a 7 point scale. 


Objective(s) of Measurement Process: Questionnaire 


quantified variables in a model, and these variables were 
used to measure attitude, quality, success of system, and 
management involvement. 


Participants in Measurement Process: Managers. 


Distribution of Measuring Instruments: Not discussed. Full 
participation of 114 managers. 


Corrective Action taken to Account for Non-participants: 
Not required, full participation. 


Identification of Differences between Participants and Non- 
participants: Not required, full participation. 


Method Used to Analyze Data: Using BMDP statistical 
computer, correlation and regression methods were used. 


Issue of Validity Addressed: Generalized the results of his 
study. 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Ordinal. 
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Decker, J. E. and J. P. Plumlee, "Microcomputers, Manpower 
and Performance in a Public Utility," Public Productivity 
Review, Vol. 9, No. 2-3, (Summer/Fall 1985), pp. 213-225. 


Description of System Studied: A work force management 
system, operated by the Jacksonville Electric Authority. 


Data Collection Technique(s) Used: Analysis of existing 


documents and reports. 


Characteristics of Measuring Instruments Used: Data base 
information was analyzed as existing documents and reports. 


Objective(s) of Measurement Process: Measurement of 


performance. 


Participants in Measurement Process: Case study of one 
utility company’s experience. 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Descriptive statistics. 


Issue of Validity Addressed: Generalized results. Implied 
external validity. Did not address internal validity. 


Type of Validity Claimed: External (implied). 


Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Metric, Interval, and Frequency. 
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Englander, V. and F. Englander, "Cost-Effectiveness of 
Computer-Based Welfare Fraud Detection in New Jersey," 
Public Productivity Review, Vol. 9, No. 2-3, (Summer/Fall 
1985), pp. 271-282. 


Description of System Studied: Software program which was 
used to detect the fraudulent receipt of public assistance 
benefits by participants in the Aid to Families with 
Dependent Children, the Food Stamps, and the Medicaid 
programs of New Jersey. 


Data Collection Technique(s) Used: Analysis of existing 


records and documents. 


Characteristics of Measuring Instruments Used: Computer 
records were analyzed quarterly to match SSN of welfare 


recipients against employed taxpayers. 


Objective(s) of Measurement Process: Data were used to 


measure cost savings. The value was determined by keeping a 
running total of reductions in benefits directly from 
closures and grant reductions detected by computer matches. 


Participants in Measurement Process: New Jersey Division of 
Public Welfare. 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Assumed full participation. 


Identification of Differences between Participants and Non- 
participants: Assumed full participation. 


Method Used to Analyze Data: Regression analysis. 
Issue of Validity Addressed: Addressed possible problems 


with the internal validity. External validity was not 
discussed. 


Type of Validity Claimed: Not discussed. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Interval. 
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Sullivan, R. S. and S. C. Secrest, "A Simple Optimization 
DSS for Production Planning at Dairyman’s Cooperative 
Creamery Association," Interfaces, Vol. 15, No. 5, 
(September-October 1985), pp. 46-53. 


Description of System Studied: A small optimization-based 
decision support system that was used to perform milk flow 
analysis at Dairyman’s Cooperative Creamery Association. 


Data Collection Technique(s) Used: Analysis of existing 


documents and reports. 


Characteristics of Measuring Instruments Used: Reports 
generated by the milk flow analysis program were the 


existing documents and reports analyzed. 


Objective(s) of Measurement Process: Reports generated by 


the milk flow program were analyzed to measure system 
performance. 


Participants in Measurement Process: Several plant 
supervisors who had been involved with production planning 


for more than 20 years. 


Distribution of Measuring Instruments: Not discussed. 


Response rate was not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Not discussed. 
Issue of Validity Addressed: Not discussed. 
Type of Validity Claimed: Not discussed. 


Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Interval and metric. 
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Lincoln, T., "Do Computer Systems Really Pay-off?," 
Information & Management, Vol. 11, No. 1, (August 1986), pp. 
25-34. 


Description of System Studied: Systems consisting of 167 
individual applications, ran by 20 companies. These 


applications reside on integrated systems, stand-alone 
systems, and general facility/service systems. 


Data Collection Technique(s) Used: Analysis of existing 
documents and records, interview, and questionnaire. 


Characteristics of Measuring Instruments Used: Existing 


documents and records were archival data such as accounting 
reports. Each interview was customized and incorporates a 

questionnaire. The participant reviewed and signed-off on 

the results of the interview. 


Objective(s) of Measurement Process: Analyses of existing 


documents and records were a measure of costs. Interviews 
and questionnaires were used to identify (measure) benefits. 


Participants in Measurement Process: Users. 
Distribution of Measuring Instruments: Distribution of 
existing documentation was not discussed. Interview and 


questionnaire appears to have been administered in person. 
Response rate was not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Descriptive statistics and 
financial analysis techniques (break-even point, mean 
benefit/cost ratio, and internal rates of return. 


Issue of Validity Addressed: The author addressed 
unreliability of user interviews. 


Type of Validity Claimed: Not discussed. 
Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Interval, Frequency, and possible 
others. 
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Franz, C.R. and D. Robey, " Organizational Context, User 
Involvement, and the Usefulness of Information Systems," 
Decision Sciences, Vol. 17, No. 3, (Summer 1986), pp. 329- 
355. 


Description of System Studied: Information systems used by 
34 organizations. The types and modes of interaction varied 
greatly between systems. 


Data Collection Technique (s) Used: Interview and 


questionnaire. 


Characteristics of Measuring Instruments Used: 
Characteristics of the interview were not discussed. A 


cross-sectional field survey. Questionnaire was Likert 
type. 


Objective(s) of Measurement Process: The interviews yielded 


a preliminary view of its computer and information system. 
Questionnaire was used to measure user’s involvement and 
perceived usefulness of system. 


Participants in Measurement Process: Initial interviews 
were with senior MIS managers. Questionnaires were 


completed by user-managers. 


Distribution of Measuring Instruments: Of the 49 managers 
contacted for interviews, 44 agreed to cooperate. 


Distribution of the questionnaire was not discussed. A 
total of 188 user-managers in 34 organizations returned the 
questionnaire. The response rate was not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Canonical correlation models 
were constructed using Statistical Package for Social 
Sciences, release 7.0 to analyze data. 


Issue of Validity Addressed: Authors stated that 
standardized instruments for measuring user involvement and 
system usefulness were unavailable at the start of the 
study. They had to develop and validate an approach to 
measure such variables. They clearly described how the 
questionnaire instrument was developed and tested. Imply 
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that the external validity of their instruments have not 
been established. 


Type of Validity Claimed: Internal (implied). 
Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Ordinal. 
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Kapsales, P., "Before and After WP: An Office Automation 
Productivity Study," Journal of Systems Management, Vol. 37, 
Ao. 4, (ApriT 1986), pp.” 7-9. 


Description of System Studied: Word processing system used 
by a major division of a State government. 


Data Collection Technique(s) Used: Analysis of existing 


documents and records. 


Characteristics of Measuring Instruments Used: Archival 
data were used as existing documents and records to collect 


data. 


Objective(s) of Measurement Process: Data were used to 


measure cost. 


Participants in Measurement Process: A major division of a 
State government. 


Distribution of Measuring Instruments: Not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Descriptive statistics. 
Issue of Validity Addressed: Presented as one-shot case 


Study, but provides a few generalizations for other 
organizations. 


Type of Validity Claimed: External (implied). 
Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Interval. 
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Snitkin, S.R. and W. R. King, "Determinants of the 
Effectiveness of Personal Decision Support Systems," 
Information & Management, Vol. 10, No. 2, (January 1986), 
pp: 83-89. 


Description of System Studied: Personal decision support 
systems. 


Data Collection Technique (s) Used: Questionnaire. 
Characteristics of Measuring Instruments Used: 


Questionnaire used a five point scale and open-ended 
questions. 


Objective(s) of Measurement Process: Questionnaire measured 


user perception of system effectiveness and the open-ended 
question allowed users to describe the most important 
applications pertaining to effectiveness. 


Participants in Measurement Process: Users of personal 
decision support systems, purchased from one vendor. 


Distribution of Measuring Instruments: Questionnaires were 
mailed to 310 users. A response rate of 18% was achieved. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: "The respondent anonymity requested by the 


computer deal precluded an analysis of non-respondents." 


Method Used to Analyze Data: Chi-square analysis. 
Issue of Validity Addressed: Not addressed. 
Type of Validity Claimed: Not discussed. 


Source of Validity Claimed: Not discussed. 


Type of Data Recorded: Qualitative, ordinal, and possible 
others. 
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Aldag, R. J. and D. J. Power, "An Empirical Assessment of 
Computer Assisted Decision Analysis," Decision Sciences, 
Varmel7, No. 14, (Fall 1986), pp. 572-588. 


Description of System Studied: DECAID, a first-generation 
computerized decision-analysis program. The program helped 
the user conduct a structured decision analysis, using 
question-and-response dialogues. 


Data Collection Technique(s) Used: Experiment and 


questionnaire. 


Characteristics of Measuring Instruments Used: Experiment 
was conducted in the laboratory. Questionnaire was Likert 


type, using five and seven point scales. 


Objective(s) of Measurement Process: Experiment was used to 


introduce post-test control group into design. The 
questionnaire measured attitudes. 


Participants in Measurement Process: Business students in 
undergraduate and graduate level normative decision-making 


courses. 


Distribution of Measuring Instruments: Assumed to have been 
distributed in class. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Variety of methods used to 
analyze correlation (Principal component factor analysis 
with varimax rotation, analysis of inter rater reliability, 
Dimensional interactive cognitive complexity, tolerance for 
ambiguity,... and one-way analysis of variance.) 


Issue of Validity Addressed: Internal validity was strongly 
implied and authors directly addressed face validity of 
program. "In conclusion, computerized decision-analysis 
aids may serve many useful functions. However, the only 
benefits clearly evident at present are that the programs 
offer students experience with a new decision technology and 
provide a structured experience in decision making. These 
programs seem to have high face validity and may result in 
positive user affect and enhanced confidence." External 
validity was addressed, "Clearly, additional studies are 
needed before confident prescriptions can be made concerning 
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the applicability and benefits of computerized decision aids 
for management decision making." 


Type of Validity Claimed: Internal. 


Source of Validity Claimed: Analysis of instrument 
measures. 


Type of Data Recorded: Ordinal. 
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Baroudi, J. J., M. H. Olson and B. Ives, "An Empirical Study 
of the Impact of User Involvement on System Usage and 
Information Satisfaction," Communications of the ACM, Vol. 
29, No. 3, (March 1986), pp. 232-238. 


Description of System Studied: A variety of information 
systems in over 200 large manufacturing firms. 


Data Collection Technique(s) Used: Questionnaire. 


Characteristics of Measuring Instruments Used: 
Questionnaire developed and partially validated by Bailey 


and Pearson. 


Objective(s) of Measurement Process: Measures of 


information satisfaction, system usage, and user 
involvement. 


Participants in Measurement Process: Production managers 
from over 200 large manufacturing firms. ' 


Distribution of Measuring Instruments: A total of 800 
questionnaires were mailed, with 200 subjects responding. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Briefly mentioned that old mailing lists were 


used, trying to account for non-response. 


Method Used to Analyze Data: Zero order correlations were 
calculated, a correlation matrix developed, and path 
coefficients calculated. 


Issue of Validity Addressed: "Data generated in this study 
were also used to further validate and modify this 
instrument." "Even though the data were collected ina 
cross-sectional survey, attention was paid to the causal 
ordering of the variables." "Generalizability across 
organizations and systems was enhanced by including over 200 
users in the same function from many different 
organizations." "Perceptual rather than objective measures 
of system usage were utilized." Authors clearly addressed 
internal and external concerns in their conclusions. 


Type of Validity Claimed: None. 


Source of Validity Claimed: Analysis of instrument 
measures. 
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Type of Data Recorded: Ordinal. 


Secondary Source(s) Cited: Bailey and Pearson (1983) is 
where the development of the original instrument was 
presented. Its validity was discussed, claiming content, 
construct, and external validity. The source of the claim 
was an analysis of the instruments measurements. 
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Franz, C. R., D. Robey and R. R. Koeblitz, "User Response to 
an Online Information System: A Field Experiment," 
Management Information Systems Quarterly, (March 1986), pp. 
2-42. 


Description of System Studied: Patient Care System, 
developed jointly by the Duke University Medical Center and 


IBM, as operated in a large hospital with over 400 beds, 
delivering more than 150,000 patient days of health care. 


Data Collection Technique(s) Used: Experiment and two 


questionnaires. 


Characteristics of Measuring Instruments Used: A quasi- 


experimental research design was employed in a field study, 
using a separate-sample pretest-post-test design as 
described by Campbell and Stanley. Job perceptions were 
measured using Job Characteristics Inventory developed by 
Sims, Szilagyi, and Keller. User attitudes were measured 
using five point Likert type instrument developed by Schultz 
and Slevin. 


Objective(s) of Measurement Process: Questionnaire used to 
collect perceptual and attitudinal data. 


Participants in Measurement Process: Nurses. 


Distribution of Measuring Instruments: Random selection of 
participating nurses with 258 completing the pretest 


questionnaire and 218 completing the post-test 
questionnaire. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Dropouts from experiment were studied to make 


suggestions regarding differences between participants and 
non-participants. 


Method Used to Analyze Data: Data were analyzed using the 
Statistical Package for the Social Sciences Version H, 
Release 9.0. 


Issue of Validity Addressed: "The design (experiment) did, 
however, permit an assessment of the effects of the pretest 
on post-test scores, a common threat to internal validity in 
experimental design." Authors addressed external validity. 


Type of Validity Claimed: None. 
141 


Source of Validity Claimed: Analysis of instrument 
measures. 


Type of Data Recorded: Ordinal. 


Secondary Source(s) Cited: Schultz and Slevin (1975) 
presented the original instrument, however, its validity was 
not clearly established. 


Sims, Szilagyi, and Keller (1976) presented analysis of 
original instrument (Job Characteristic Inventory) and 
claimed to have demonstrated construct and discriminant- 
convergent validity. The source of the claim was an 
analysis of the instruments measures. 
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Durand, D. E., R. O. Bennett and S. Betty, "What Did 
Information Technology ’Do’ to Business Communications?: 
Two Empirical Studies," Information & Management, Vol. 13, 
No. 3, (October 1987), pp. 111-117. 


Description of System Studied: Enable, a word processing 
software package. 


Data Collection Technique(s) Used: Experiment using 


participant observation. 


Characteristics of Measuring Instruments Used: Observations 
of written output and speed of reading were recorded and 


analyzed. 


Objective(s) of Measurement Process: Observations measured 


writing style and content and speed of reading. 


Participants in Measurement Process: Students in an 
introductory information systems course were observed to 


measure writing style and content. Business professionals 
were observed to measure speed of reading. 


Distribution of Measuring Instruments: A total of 79 
students participated. It was assumed that the observations 


were made in the student’s class. Full participation of 
students is assumed. Distribution to business professionals 
was not discussed, nor were their response rates. 


Corrective Action taken to Account for Non-participants: 
Assumed full participation of students. Not discussed with 


business professionals. 


Identification of Differences between Participants and Non- 
participants: Assumed full participation of students. Not 


discussed with business professionals. 


Method Used to Analyze Data: Observations (the written 
output of students) was analyzed using Quintilian Analysis, 
a computer program specifically designed to analyze writing 
style and content. Non-parametric statistical procedures 
(Mann-Whitney U for ordinal data) and parametric statistics 
for ratio data. Descriptive statistics were used to analyze 
the reading speed data. 


Issue of Validity Addressed: Not discussed. Internal and 
external validity are implied in findings. 
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Type of Validity Claimed: Internal (implied) and External 
(implied). 


Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Interval and frequency. 
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Gallupe, R. B. and G. DeSanctis, "Computer-Based Support for 
Group Problem Finding: An Experimental Investigation," 
Management Information System Quarterly, Vol. 12 , No. 2, 
(June 1988), pp. 277-296. 


Description of System Studied: DECAID, a group decision 
support system. 


Data Collection Technique(s) Used: Experiment using a panel 


of three judges to compile observations, and two 
questionnaires. 


Characteristics of Measuring Instruments Used: Experiment 
was of pre and post test design. The panel of three judges 


apparently assumed the role of complete observers; 
performing document analysis. All questionnaires used a 
seven point Likert scale. Interviews were conducted in a 
group setting. 


Objective(s) of Measurement Process: Observations made by 


the three judges were measures of decision quality. The 
questionnaires and interviews measured group members' 
perceptions of group decision confidence and group 
satisfaction. 


Participants in Measurement Process: Senior undergraduate 
business administration students. Random assignment of 


respondents to various experiment groups. 


Distribution of Measuring Instruments: A total of 84 
students participated in the study. No discussion of 


response rate was provided, it is assumed that this was part 
of a class assignment. All questionnaires and interviews 
were administered in person. 


Corrective Action taken to Account for Non-participants: 
Assumed full participation. 


Identification of Differences between Participants and Non- 
participants: Assumed full participation. 


Method Used to Analyze Data: A fixed-effect two-way 
analysis of a variance model was applied to all dependent 
measures. 
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Issue of Validity Addressed: "All of these findings must be 


interpreted with caution, given the limited scope and 
sampling used in this study. As a controlled experimental 
study the findings cannot necessarily be generalized to all 
levels or types of GDSS software or to actual organizational 
use of GDSS technology." 


Type of Validity Claimed: Internal (implied). 


Source of Validity Claimed: Not discussed. 
Type of Data Recorded: Qualitative and ordinal. 
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Tait, P. and I. Vessey, "The Effect of User Involvement on 
System Success: A Contingency Approach" Management 
Information System Quarterly, (March 1988), pp. 91-108. 


Description of System Studied: 42 recently implemented 
custom-built information systems, within Australian firms. 


Data Collection Technique(s) Used: Questionnaires. 


Characteristics of Measuring Instruments Used: All 
questionnaires were either off-the-shelf or employed 


techniques used in previous studies. 


Objective(s) of Measurement Process: Questionnaires 
administered to the users measured success of the system, 
user involvement in design, impact of the system on the 
organization, and the attitudes of the users. 
Questionnaires administered to the designers measured the 
technical complexity of the system and the resource 
constraints on the development of the system. 


Participants in Measurement Process: Users and designers of 
the information systems. 


Distribution of Measuring Instruments: A total of 84 system 
users and designers participated in the study, achieving a 


71 percent response rate. Questionnaires were sent by mail. 


Corrective Action taken to Account for Non-participants: 
Follow-up phone calls. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Path analysis. 


Issue of Validity Addressed: "Data for the study were 
collected after the systems were implemented. Subjects’ 
responses may have been influenced by the ultimate success 
or failure of the systems. The timing of the data 
collection may have had a secondary influence on the 
measurement of user attitudes.... A further limitation of 
the study is the use of instruments that are not 
sufficiently validated--the perennial problem of a new 
discipline." 


Type of Validity Claimed: None. 


Source of Validity Claimed: None. 
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Type of Data Recorded: Ordinal and possible others. 


Secondary Source(s) Cited: Questionnaire was comprised of 
subsections, using instruments developed by Wolek (1975), 
Elizur and Guttman (1976), Ein-Dor and Segev (1978), Keen 
(1981), McFarlan (1981), Ives, Olson, and Baroudi (1983). 


Discussions of validity were presented in Elizur and Guttman 
(1976) and Ives et al. (1983). None of the other 


discussions significantly substantiate the validity of this 
study. 
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Cronan, T. P. and D. E. Douglas, "End-User Training and 
Computing Effectiveness in Public Agencies: An Empirical 
Study." Journal of Management Information Systems, Vol. 6, 
NA (Spring T990), pp. 21-39. 


Description of System Studied: End user computing systems 
of an overall public agency involved in financial services. 


Data Collection_Technique (s) Used: Two questionnaires and 


analysis of existing documents and records. 


Characteristics of Measuring Instruments Used: 
Questionnaire administered to users completing the training 


program used a five point scale (Not focus of study). 
Separate questionnaires administered to end users and 
supervisors use a five point type scale and are shortened 
versions of previously used instruments in other studies. 
Existing documents and records were agency statistics. 


Objective(s) of Measurement Process: Questionnaires 


measured training program, EUC effectiveness, and end user 
acceptance/satisfaction. 


Participants in Measurement Process: Over 100 public 
personnel, predominately accountants. Users completing the 


training program evaluated the training program. End users 
and supervisors measured end user computing effectiveness 
and end user acceptance/satisfaction. 


Distribution of Measuring Instruments: A total of 39 
responses were used to evaluate the training program, 


distribution and response rate was not discussed. 


Corrective Action taken to Account for Non-participants: 
Not discussed. 


Identification of Differences between Participants and Non- 
participants: Not discussed. 


Method Used to Analyze Data: Paired t-test statistic, Chi 
Square test statistic, partied t-test statistic, Wilcoxon 
Signed rank test statistic. 


Issue of Validity Addressed: Explained types of validity 
and presented argument that sufficient degree of correlation 


exist to claim reliability, convergent, and construct 
validity. Advised caution should be exercised in complete 
generalizations of the results. 


Type of Validity Claimed: Convergent and construct. 
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Source of Validity Claimed: Results of correlation 
analysis. 


Type of Data Recorded: Ordinal, interval, frequency, and 
possible others. 


Secondary Source(s) Cited: Baroudi and Orlikowski (1988), 
Doll and Torkzadeh (1988), and Ives, Olson, and Baroudi 
(1983) are where the development of the original instruments 
were presented. Its validity was discussed by all the above 
authors, claiming construct, convergent-discriminant, and 
criterion validity. The source of the claims are analyses 
of measurements from the instrument. 
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APPENDIX C 


CONSOLIDATED ANALYSIS OF STUDIES 


(Follows on succeeding pages.) 
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